Image processing and model training method, device, equipment and storage medium

Abstract

The present application provides an image processing and model training method, device, equipment and storage medium, which can be applied to various fields such as AI drawing and model training. The method includes: obtaining a first image and a prompt word, the prompt word is used to indicate the target live attack type; extracting the image feature information of the first image and the text feature information of the prompt word, determining the target reference feature information, the target reference feature information is used to represent the image features of the target live attack, and then using the text feature information of the prompt word and the target reference feature information as control conditions, the image feature information of the first image is processed through the image generation model to obtain the first live attack image. That is, the present application uses the target reference feature information and the prompt word as control conditions to process the first image, and the generated first live attack image includes not only the features described by the prompt word, but also the image features of the target live attack, thereby improving the generation quality of the first live attack image.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer technology, and in particular to an image processing and model training method, apparatus, device and storage medium. Background Art

[0002] With the rapid development of artificial intelligence (AI) technology, AI drawing applications have emerged. AI drawing includes two application modes: text-to-image and image-to-image. During the model training process, training images can be generated through AI drawing. For example, during the training of a liveness detection model, the types of liveness attacks are so complex and numerous that human data collection and annotation alone cannot meet actual business needs and cannot construct effective negative samples. At this time, AI drawing can be used to generate training images corresponding to the specified liveness attack type.

[0003] However, currently when generating another image based on one image, the quality of the generated image is not ideal and cannot meet the expected requirements. Summary of the invention

[0004] The present application provides an image processing and model training method, apparatus, device and storage medium, which can improve the quality of image generation.

[0005] In a first aspect, the present application provides an image processing method, comprising:

[0006] Acquire a first image and a prompt word, wherein the prompt word is used to indicate a target live body attack type, and the first image is a biometric feature image;

[0007] Extracting image feature information of the first image and text feature information of the prompt word;

[0008] Determining target reference feature information, where the target reference feature information is used to represent image features of a target live attack, where the target live attack belongs to the target live attack type;

[0009] The text feature information of the prompt word and the target reference feature information are used as control conditions, and the image feature information of the first image is processed by an image generation model to obtain a first live attack image for the biometric feature image.

[0010] In some embodiments, before extracting the image feature information of the first image, the method further includes:

[0011] Performing a cutout process on the first image to cut out an object region in the first image;

[0012] adjusting the size of the object area to a first size;

[0013] Performing center cropping on the object region of the first size to a second size to obtain the preprocessed first image;

[0014] The extracting the image feature information of the first image includes:

[0015] Image feature extraction is performed on the preprocessed first image to obtain image feature information of the first image.

[0016] In a second aspect, the present application provides a model training method, comprising:

[0017] Acquire a training image and a prompt word, wherein the prompt word is used to indicate the target live attack type, and the training image is a biometric image or a live attack image;

[0018] Extracting image feature information of the training image and text feature information of the prompt word;

[0019] Determine target reference feature information, where the target reference feature information is used to represent target live attack features, and the target live attack belongs to the target live attack type;

[0020] Using the text feature information of the prompt word and the target reference feature information as control conditions, the image feature information of the training image is processed by an image generation model to obtain a third live attack image for the biometric feature image;

[0021] Based on the target reference feature information and the third living body attack image, a loss of the image generation model is determined, and the image generation model is trained based on the loss.

[0022] In some embodiments, determining the loss of the image generation model based on the texture feature information of the training image and the texture feature information of the third living attack image includes:

[0023] Determine a difference between the texture feature information of the training image and the texture feature information of the third living body attack image;

[0024] Based on the difference, a loss of the image generation model is determined.

[0025] In some embodiments, before extracting the image feature information of the training image, the method further includes:

[0026] Performing a cutout process on the training image to cut out an object region in the training image;

[0027] adjusting the size of the object area to a first size;

[0028] Randomly cropping the object region of the first size to a second size to obtain the preprocessed training image;

[0029] The extracting the image feature information of the training image comprises:

[0030] Image feature extraction is performed on the preprocessed training image to obtain image feature information of the training image.

[0031] In a third aspect, the present application provides an image processing device, comprising:

[0032] An acquisition unit, used for acquiring a first image and a prompt word, wherein the prompt word is used for indicating a target live body attack type, and the first image is a biometric feature image;

[0033] A feature extraction unit, used to extract image feature information of the first image and text feature information of the prompt word;

[0034] a determination unit, configured to determine target reference feature information, wherein the target reference feature information is used to represent image features of a target live attack, and the target live attack belongs to the target live attack type;

[0035] The processing unit is used to process the image feature information of the first image through an image generation model using the text feature information of the prompt word and the target reference feature information as control conditions to obtain the first live attack image for the biometric feature image.

[0036] In some embodiments, the determination unit is specifically used to determine information of N candidate reference features corresponding to the target live attack type, and select information of a candidate reference feature from the information of the N candidate reference features as the target reference feature information, where the information of the N candidate reference features is image feature information of N types of live attacks belonging to the target live attack type, and N is a positive integer; or, obtain a second live attack image, and extract live attack feature information of the second live attack image as the target reference feature information.

[0037] In some embodiments, the information of the N candidate reference features or the live attack feature information of the second live attack image is image texture feature information.

[0038] In some embodiments, the image generation model includes a denoising network and a processing unit, which is specifically used to perform noise processing on the image feature information of the first image to obtain the noisy feature information of the first image; using the text feature information of the prompt word and the target reference feature information as control conditions, the noisy feature information of the first image is denoised through the denoising network to obtain the denoised image feature information; and the denoised image feature information is decoded to obtain the first living attack image.

[0039] In some embodiments, the processing unit is specifically used to use the text feature information of the prompt word and the target reference feature information as control conditions, and perform denoising on the i-th denoising feature information of the first image through the denoising network to obtain the i-th noise prediction value, where i is a positive integer less than or equal to N. If i is 1, the i-th denoising feature information is the noisy feature information of the first image; based on the i-th noise prediction value and the i-th denoising feature information of the first image, obtain the i+1-th denoising feature information of the first image; use the i+1-th denoising feature information as the new i-th denoising feature information, return and repeat the above steps to obtain the N-th denoising feature information of the first image; determine the N-th denoising feature information as the denoised image feature information.

[0040] In some embodiments, the processing unit is specifically used to use the i-th denoising feature information of the first image as the input feature information of the denoising network; fuse the feature information output by some network layers of the denoising network with at least the text feature information of the prompt word, and fuse the feature information output by some network layers of the denoising network with at least the target reference feature information to obtain the i-th noise prediction value.

[0041] In some embodiments, the processing unit is specifically used to fuse the output feature information of the kth network layer in the denoising network with the text feature information of the prompt word and at least one of the target reference feature information to obtain the input feature information of the k+1th network layer in the denoising network, where k is a positive integer; and obtain the i-th noise prediction value based on the input feature information of the k+1th network layer.

[0042] In some embodiments, the processing unit is specifically used to fuse the output feature information of the kth network layer with the text feature information of the prompt word and at least one of the target reference feature information through an attention mechanism to obtain the input feature information of the k+1th network layer.

[0043] In some embodiments, the processing unit is specifically used to use the text feature information of the prompt word as the key value of the attention module, use the target reference feature information as the representative value of the attention module, and use the output feature information of the kth network layer as the query value of the attention module, input it into the attention module for fusion processing, and obtain the input feature information of the k+1th network layer.

[0044] In some embodiments, the feature extraction unit is specifically used to preprocess the first image to obtain a preprocessed first image; and extract image features from the preprocessed first image to obtain image feature information of the first image.

[0045] In some embodiments, the feature extraction unit is specifically used to perform cutout processing on the first image to cut out the object area in the first image; adjust the size of the object area to a first size; center-crop the object area of ​​the first size to a second size, so as to obtain the preprocessed first image.

[0046] In some embodiments, the first image and the first live attack image are used as training images for a target model.

[0047] In a fourth aspect, the present application provides a model training device, comprising:

[0048] An acquisition unit, used for acquiring a training image and a prompt word, wherein the prompt word is used for indicating a target live attack type, and the training image is a biometric image or a live attack image;

[0049] A feature extraction unit, used to extract image feature information of the training image and text feature information of the prompt word;

[0050] a determination unit, configured to determine target reference feature information, wherein the target reference feature information is used to represent a target live body attack feature, and the target live body attack belongs to the target live body attack type;

[0051] a processing unit, configured to process the image feature information of the training image by using the text feature information of the prompt word and the target reference feature information as control conditions through an image generation model to obtain a third live attack image for the biometric feature image;

[0052] A training unit is used to determine the loss of the image generation model based on the target reference feature information and the third living attack image, and train the image generation model based on the loss.

[0053] In some embodiments, the determination unit is specifically used to determine information of N candidate reference features corresponding to the target live attack type, and select information of one candidate reference feature from the information of the N candidate reference features as the target reference feature information, where the information of the N candidate reference features is image feature information of N types of live attacks belonging to the target live attack type, and N is a positive integer; or, if the training image is a live attack image, extract the live attack feature information of the training image as the target reference feature information; or, obtain a fourth live attack image, extract the live attack feature information of the fourth live attack image as the target reference feature information.

[0054] In some embodiments, the training unit is specifically configured to perform live attack feature extraction on the third live attack image to obtain live attack feature information of the third live attack image; and determine the loss of the image generation model based on the target reference feature information and the live attack feature information of the third live attack image.

[0055] In some embodiments, if the target reference feature information is the texture feature information of the training image, the training unit is specifically used to extract texture features of the third live attack image to obtain texture feature information of the third live attack image; based on the texture feature information of the training image and the texture feature information of the third live attack image, determine the loss of the image generation model.

[0056] In some embodiments, the training unit specifically determines a difference between the texture feature information of the training image and the texture feature information of the third living attack image; and determines a loss of the image generation model based on the difference.

[0057] In some embodiments, the feature extraction unit is specifically used to preprocess the training image to obtain a preprocessed training image; and extract image features from the preprocessed training image to obtain image feature information of the training image.

[0058] In some embodiments, the feature extraction unit is specifically used to perform cutout processing on the training image to cut out the object area in the training image; adjust the size of the object area to a first size; randomly crop the object area of ​​the first size to a second size to obtain the preprocessed training image.

[0059] In a fifth aspect, the present application provides an electronic device, comprising a processor and a memory, wherein the memory is used to store a computer program, and the processor is used to call and run the computer program stored in the memory to execute the method of the first aspect or the second aspect.

[0060] In a sixth aspect, a chip is provided for implementing the method in each implementation of the first aspect. Specifically, the chip includes: a processor for calling and running a computer program from a memory, so that a device equipped with the chip executes the method in the first or second aspect.

[0061] In a seventh aspect, a computer-readable storage medium is provided for storing a computer program, wherein the computer program enables a computer to execute the method of the first aspect or the second aspect.

[0062] In an eighth aspect, a computer program product is provided, comprising computer program instructions, wherein the computer program instructions enable a computer to execute the method of the first aspect or the second aspect.

[0063] In a ninth aspect, a computer program is provided, which, when executed on a computer, enables the computer to execute the method of the first aspect or the second aspect.

[0064] In summary, the present application obtains a first image and a prompt word, the prompt word is used to indicate the target live attack type, and the first image is a biometric image; performs image feature extraction on the first image to obtain image feature information of the first image, and performs text feature extraction on the prompt word to obtain text feature information of the prompt word, and determines target reference feature information, the target reference feature information is used to indicate the image features of the target live attack, and the target live attack belongs to the target live attack type. Then, the text feature information of the prompt word and the target reference feature information are used as control conditions, and the image feature information of the first image is processed through the image generation model to obtain a first live attack image for the biometric image. It can be seen that when the embodiment of the present application generates the first live attack image based on the first image and the prompt word, the target reference feature information and the prompt word used to represent the image features of the target live attack are used as control conditions to process the first image and generate the first live attack image, so that the generated first live attack image not only meets the requirements of the target live attack type indicated by the prompt word, but also includes the image features of the target live attack under the target live attack type, so that the first live attack image includes more image details of the target live attack type, thereby improving the generation quality of the first live attack image. If the above image processing method is applied to the training data for generating a liveness detection model, a realistic and high-quality liveness attack sample can be generated. When the liveness detection model is trained using the realistic and high-quality liveness attack sample, the liveness detection model can be improved to accurately identify realistic liveness attack data, thereby improving the detection effect of the liveness detection model. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1A schematic diagram of an implementation environment of an image processing method provided in an embodiment of the present application;

[0066] Figure 2 A flowchart of a model training method provided in one embodiment of the present application;

[0067] Figure 3 It is a schematic diagram of a graph;

[0068] Figure 4 It is a schematic diagram of extracting image feature information;

[0069] Figure 5 It is a schematic diagram of extracting text feature information;

[0070] Figure 6 A schematic diagram of an image processing process involved in an embodiment of the present application;

[0071] Figure 7 A schematic diagram of an image generation model;

[0072] Figure 8 A schematic diagram of denoising using a denoising network;

[0073] Fig. 9 A structural diagram of a denoising network;

[0074] Fig.10 Schematic diagram of data processing between network layers in the denoising network;

[0075] Fig.11 , Fig.12 , Fig.13 A schematic diagram of fusing the output feature information of the network layer of the denoising network with the text feature information of the prompt word and the target reference feature information;

[0076] Fig.14A , Fig. 14B , Fig. 14C A schematic diagram of fusing the kth network layer in the denoising network with at least one of the text feature information of the prompt word and the target reference feature information;

[0077] Fig.15 A specific network diagram of a denoising network;

[0078] Fig.16 A schematic diagram of using a decoder to generate a third live attack image;

[0079] Fig.17 A schematic diagram of the image processing method flow provided by an embodiment of the present application;

[0080] Fig.18 A schematic diagram of an image generation process;

[0081] Fig.19 A flowchart of a training sample generation method provided in an embodiment of the present application;

[0082] Fig. 20 A schematic diagram of a training sample generation process;

[0083] Fig.21 is a schematic block diagram of an image processing device provided by an embodiment of the present application;

[0084] Fig. 22 is a schematic block diagram of a model training device provided in one embodiment of the present application;

[0085] Fig.23 It is a schematic block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0086] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0087] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described here can be implemented in an order other than those illustrated or described here. In an embodiment of the present invention, "B corresponding to A" means that B is associated with A. In one implementation, B can be determined according to A. However, it should also be understood that determining B according to A does not mean that B is determined only according to A, but B can also be determined according to A and / or other information. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or server that includes a series of steps or units does not have to be limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices. In the description of the present application, unless otherwise specified, "multiple" refers to two or more than two.

[0088] The image processing method provided in the embodiment of the present application can be applied to various fields such as AI drawing and model training. When generating a first live attack image based on a first image, the prompt word and target reference feature information used to represent the image features of the target live attack can be used as control information to process the first image to generate the first live attack image. The generated first live attack image not only meets the requirements of the prompt word, but also meets the image features of the target live attack, thereby improving the generation quality of the first live attack image.

[0089] In order to facilitate understanding of the embodiments of the present application, the relevant concepts involved in the embodiments of the present application are first introduced:

[0090] Image generation refers to transforming the features of the object in the initial image according to the input prompt word to generate a target image that meets the prompt word requirements. At present, the image generation method is usually generated based on the image generation model of the diffusion algorithm, specifically transforming the initial image according to the prompt word input by the object.

[0091] The image processing of the embodiments of the present application can be applied to any scenario where an image is generated from another image, that is, it can be applied to a scenario where another image is generated based on one image.

[0092] In some embodiments, the image processing method of the embodiment of the present application can be used to generate training samples. For example, to generate training samples for a liveness detection model. Liveness detection technology is applied in many fields, such as smart face payment services, identity authentication services, etc. Due to the complexity and variety of liveness attack types, manpower alone for data collection and labeling cannot meet actual business needs. The collected data may be homogenized and single, and it is impossible to construct effective negative samples (ie, bad cases), and then the liveness detection model is not thoroughly trained, reducing the detection accuracy of the liveness detection model, and then there is a risk of infringement. The methods for generating liveness attack samples mainly include methods based on traditional GAN ​​(Generative Adversarial Networks) and methods based on diffusion models.

[0093] Among them, the main idea of ​​the live attack sample generation technology based on traditional GAN ​​is to randomly initialize a Gaussian noise RGB image, input the Gaussian noise image into the encoder part of the generator, obtain the corresponding encoded feature vector, and then input the feature vector into the decoder part of the generator, and obtain an image of the same size and dimension as the original Gaussian noise RGB image by upsampling. Then, the image and the real live attack image (such as 3D mask, head model, paper copy or screen copy image, etc.) are input into a binary classification network for true and false discrimination. The generator ensures that the third image is as similar to the real live attack image as possible, while the discriminator gives a fake label to the generated image and a real live attack image a real label. Through this continuous adversarial training method, the generator can eventually generate results similar to real live attack images for any input Gaussian noise image. However, the overall performance of the live attack sample generation technology based on traditional GAN ​​is poor, and the quality of the third image is poor.

[0094] The live attack sample generation technology based on the diffusion model generates the corresponding graph by taking the text information as the control condition. However, the live attack sample generation technology based on the diffusion model generates live attack samples that are not realistic enough and of poor quality, and thus cannot meet the training requirements, resulting in poor model training results.

[0095] In order to solve the above technical problems, the embodiment of the present application proposes an image processing method, which obtains a first image and a prompt word, the prompt word is used to indicate the target live attack type, and the first image is a biometric image; image feature extraction is performed on the first image to obtain image feature information of the first image, and text feature extraction is performed on the prompt word to obtain text feature information of the prompt word, and target reference feature information is determined, and the target reference feature information is used to indicate the image feature of the target live attack, and the target live attack belongs to the target live attack type. Then, the text feature information of the prompt word and the target reference feature information are used as control conditions, and the image feature information of the first image is processed through an image generation model to obtain a first live attack image for the biometric image. It can be seen that when the embodiment of the present application generates the first live attack image based on the first image and the prompt word, the target reference feature information and the prompt word used to represent the image features of the target live attack are used as control conditions to process the first image and generate the first live attack image, so that the generated first live attack image not only meets the requirements of the target live attack type indicated by the prompt word, but also includes the image features of the target live attack under the target live attack type, thereby improving the generation quality of the first live attack image. If the above image processing method is applied to the training data for generating a liveness detection model, realistic and high-quality liveness attack samples can be generated. When the liveness detection model is trained using the realistic and high-quality liveness attack samples, the liveness detection model can be improved to accurately identify realistic liveness attack data, thereby improving the detection effect of the liveness detection model.

[0096] The implementation environment of the image processing method provided in the embodiment of the present application is introduced below.

[0097] Figure 1 A schematic diagram of an implementation environment of an image processing method provided in an embodiment of the present application, such as Figure 1 As shown, the implementation environment includes: a terminal device 101 and a server 102.

[0098] The terminal device 101 is connected to the server 102 via a wired or wireless connection.

[0099] In some embodiments, the terminal device 101 is installed with a client of the image processing system, and the server 102 can be understood as the server or backend of the image processing system. An object (such as a user) can interact with the client of the image processing system installed on the terminal device 101. The server 102 stores an image generation model, which can be, for example, an image generation model based on a diffusion algorithm.

[0100] In some embodiments, before processing the image, the embodiment of the present application first trains the image generation model. Here, the process of training the image generation model by the server 102 is introduced. Specifically, the object can send a prompt word to the server 102 through a terminal device, and the prompt word is used to indicate the control information when the first live attack image is generated based on the first image. The server 102 obtains the training image from the database, or the terminal device sends the training image to the server. Then, the server 102 extracts the image feature of the training image to obtain the image feature information of the training image, and extracts the text feature of the prompt word to obtain the text feature information of the prompt word, and the server 102 determines the target reference feature information, which is used to represent the target live attack feature. Then, the server 102 uses the text feature information of the prompt word and the target reference feature information as control conditions, processes the image feature information of the training image through the image generation model, and obtains a third live attack image; based on the training image and the third live attack image, the loss of the image generation model is determined, and the image generation model is trained based on the loss. For example, the server 102 determines a difference between the target reference feature information and the live attack feature information of the third live attack image, and determines a loss of the image generation model based on the difference.

[0101] In some embodiments, the above-mentioned image generation model can also be trained through the terminal device 101, and the embodiments of the present application do not limit this.

[0102] In an embodiment of the present application, after the image generation model is trained, the trained image generation model is used to execute the image processing method of the embodiment of the present application.

[0103] In some embodiments, the image processing method provided by the embodiment of the present application is jointly executed by the terminal device 101 and the server 102. For example, the object starts the client of the image processing system on the terminal device 101, and the client displays the input interface of the first image and the input box of the prompt word. The object enters the first image to be processed in the input interface and enters the prompt word in the prompt word input box. The terminal device 101 sends the first image and the prompt word to the server 102. The server 102 extracts the image feature information of the first image and the text feature information of the prompt word. At the same time, the server 102 determines the target reference feature information, which is used to represent the target live attack feature. Then, the server 102 uses the text feature information of the prompt word and the target reference feature information as control conditions, and processes the image feature information of the first image through the image generation model to obtain the first live attack image. For example, the server 102 includes an image generation model, and the server 102 inputs the image feature information of the first image, the text feature information of the prompt word, and the target reference feature information into the image generation model to generate the first live attack image. Finally, the server 102 sends the generated first live attack image to the terminal device 101, and the terminal device 101 displays the first live attack image to the object. It can be seen that when the embodiment of the present application generates the first live attack image based on the first image and the prompt word, the target reference feature information and the prompt word are used as control conditions to process the first image to generate the first live attack image. The generated first live attack image not only includes the features described by the prompt word, but also includes the target live attack features, thereby improving the generation quality of the first live attack image. If the above image processing method is applied to the training data for generating a liveness detection model, a realistic, high-quality liveness attack sample can be generated. When the liveness detection model is trained using the realistic, high-quality liveness attack sample, the liveness detection model can be improved to accurately identify realistic liveness attack data, thereby improving the detection effect of the liveness detection model.

[0104] In some embodiments, the image processing method provided in the embodiments of the present application is executed by the terminal device 101. At this time, the terminal device 101 can be any terminal device that can provide image processing services. The server 102 can be any server that can provide data download services. In one example, the image generation model is stored in the above-mentioned server 102. A client of the image processing system is installed on the terminal device 101, and the client can request the image generation model from the server 102 based on the request of the object. For example, the terminal device 101 displays an input interface for the first image and an input box for the prompt word. The object inputs the first image to be processed in the input interface and inputs the prompt word in the input box of the prompt word. In response to the input operation of the first image and / or the prompt word, the terminal device 101 sends a request to the server 102 to request the image generation model. The terminal device 101 performs image feature extraction on the first image to obtain image feature information of the first image, and performs text feature extraction on the prompt word to obtain text feature information of the prompt word. At the same time, the terminal device 101 determines target reference feature information, which is used to represent the target live attack feature. Then, the text feature information of the prompt word and the target reference feature information are used as control conditions, and the image feature information of the first image is processed through the image generation model, so that the generated first live attack image not only meets the requirements of the prompt word, but also includes the target live attack feature, thereby improving the image generation effect.

[0105] In some embodiments, the terminal device 101 includes but is not limited to: desktop computers, laptop computers, smart phones, tablet computers, IoT devices, and portable wearable devices. IoT devices can be smart speakers, smart TVs, smart air conditioners, smart car devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. Terminal devices are often equipped with display devices, which can also be displays, display screens, touch screens, etc., and touch screens can also be touch screens, touch panels, etc.

[0106] In some embodiments, the above-mentioned server 102 may be one or more. When there are multiple servers, there are at least two servers for providing different services, and / or there are at least two servers for providing the same service, such as providing the same service in a load balancing manner, which is not limited in the embodiments of the present application. Among them, the above-mentioned server may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. The server can also become a node of the blockchain.

[0107] It should be noted that the implementation environment of the embodiments of the present application includes but is not limited to Figure 1 shown.

[0108] The technical solutions of the embodiments of the present application are described in detail below through some embodiments. The following embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.

[0109] First combine Figure 2 , the training process of the image generation model of the embodiment of the present application is introduced.

[0110] Figure 2 The flowchart of the model training method provided in one embodiment of the present application is shown in FIG. The execution subject of the embodiment of the present application may be a device having a model training function, such as a model training device. The model training device may be the above-mentioned Figure 1 The server 102 or terminal device 101 shown may also be Figure 1 The system is composed of the server 102 or the terminal device 101. For the convenience of description, the following embodiments take electronic devices as an example to illustrate the method of the embodiments of the present application.

[0111] like Figure 2 As shown, the model training method of the embodiment of the present application includes:

[0112] S101, obtaining training images and prompt words.

[0113] The prompt word is used to indicate the target live attack type, and the training image is a biometric image, for example, a real biometric image, or a live attack image.

[0114] It should be noted that the data used in the embodiments of the present application and the process of obtaining these data are in compliance with the provisions of relevant laws and regulations.

[0115] The image generation model of the embodiment of the present application can be applied in different fields to generate images that meet different requirements.

[0116] In one example, the image generation model of the embodiment of the present application can be applied to the field of AI drawing to generate images that meet the requirements.

[0117] In one example, the image generation model of the embodiment of the present application can be applied to the field of training sample generation to generate training samples that meet the requirements. For example, the image generation model can be used to generate training data for a liveness detection model, that is, to generate realistic, high-quality liveness attack samples.

[0118] In the embodiment of the present application, when the image generation model is used in different fields, the image generation model can be trained using training images of the corresponding fields.

[0119] In the field of image generation, the requirements for image generation can be described by prompt words. For example, Figure 3 As shown in , the prompt word is used to indicate the generation of a kitten. In this way, in the field of image generation, the prompt word can be used to generate Figure 3 The puppy in the left image is generated Figure 3 Kitten shown on the right.

[0120] In the embodiment of the present application, the prompt word is used to indicate the target living body attack type and to transform the initial features of the object in the training image into the target features described by the prompt word.

[0121] The types of live attack in the embodiment of the present application include 3D mask, paper copy, screen copy, 3D head model and other attack types. In the embodiment of the present application, different live attack types may include different live attack features. In the embodiment of the present application, the target live attack type may be indicated by a prompt word, for example, indicating that the target live attack type is 3D mask or screen copy, etc.

[0122] In the embodiment of the present application, the electronic device can process the training image based on the target live attack type indicated by the prompt word to generate a live attack image corresponding to the target live attack type. For example, when the target live attack type indicated by the prompt word is a 3D mask, the electronic device processes the training image to generate a 3D mask image, which is a live attack image.

[0123] The embodiment of the present application does not limit the specific method in which the electronic device obtains the training image and the prompt word.

[0124] In a possible implementation, the prompt word is input by a subject (e.g., a user). The training image is training data in a database. For example, when the electronic device is a terminal device, the subject inputs the prompt word to the terminal device. For another example, when the electronic device is a server, the subject inputs the prompt word on the terminal device, and the terminal device sends the prompt word input by the subject to the server.

[0125] In a possible implementation, the above-mentioned prompt words and training images are input by the subject. For example, when the electronic device is a terminal device, the subject inputs the training image and prompt words to the terminal device. For another example, when the electronic device is a server, the subject inputs the training image and prompt words on the terminal device, and the terminal device sends the training image and prompt words input by the subject to the server. Optionally, the above-mentioned training image can be an image taken by the subject, or an image obtained by the subject through other means.

[0126] In a possible implementation, the training image is generated by an electronic device, and the prompt word is input by the object. For example, when the electronic device is a terminal device, the terminal device receives an image generation instruction input by the object, and the image generation instruction is used to instruct the generation of a training image including at least one object. The terminal device generates a training image based on the image generation instruction. For another example, when the electronic device is a server, the terminal device receives an image generation instruction input by the object, and sends the image generation instruction to the server. The server generates a training image based on the image generation instruction.

[0127] In an embodiment of the present application, the training process of the image generation model using each training image in the training data set is basically the same. For the sake of ease of description, a training image is used as an example for illustration. In an embodiment of the present application, the image generation model is trained for multiple rounds, and a batch of training data is selected for training in each round of training. The training image can be understood as a training image in any round of training data. It should be noted that in an embodiment of the present application, during multiple rounds of training of the image generation model, the prompt words corresponding to each round of training may be different, for example, different prompt words are input in each round of training, or a prompt word corresponding to each piece of training data. Optionally, the prompt words input in each round of training may be the same. The embodiment of the present application does not limit the input of the prompt words, and can be set according to actual needs.

[0128] S102: extracting image feature information of the training image and text feature information of the prompt word.

[0129] In an embodiment of the present application, after the electronic device obtains the training image and the prompt word corresponding to the training image based on the above steps, in order to achieve mutual fusion between the image information of the training image and the semantic information of the prompt word, the electronic device first extracts the image feature information of the training image and the text feature information of the prompt word.

[0130] In some embodiments, before extracting image feature information of a training image, the electronic device first preprocesses the training image to obtain a preprocessed training image.

[0131] The specific manner in which the training image is preprocessed in the embodiment of the present application is not limited, for example, the clarity and sharpness of the training image are processed.

[0132] In one example, the electronic device crops the size of the training image to be consistent with the size of the input image of the image generation model based on the size of the input image of the image generation model.

[0133] In some examples, the electronic device performs a cutout process on the training image to cut out the object region in the first image, and then adjusts the size of the object region to a first size. Then, the electronic device randomly crops the object region of the first size to a second size to obtain a preprocessed training image.

[0134] Exemplarily, assuming that the input of the image generation model is a 224*224*3 image, in order to achieve this purpose, the training images need to be resized to this scale. In addition, in the embodiment of the present application, the prompt word is used to indicate the target live attack type. Based on this, the electronic device cuts out the object area in the training image. For example, if the training image includes a face, the face detection tool can be used to perform face detection on the face area in the training image, and then the face area is cut out on the training image according to the detection result. Next, the cut-out face area is scaled (resized) to a first size, for example, to 256*256*3. Finally, the electronic device uses a random crop method to obtain an input image of a second size (for example, 224*224*3).

[0135] In the embodiment of the present application, the training image is passed through an image feature encoder to extract latent space features. The main purpose of this part is to represent the features of the image using high-level semantic features, which can not only reduce the dimension of the original image space to increase the training speed, but also facilitate better interaction with subsequent conditional features.

[0136] In some examples, the image feature encoder may be any pre-trained model.

[0137] In one example, the image feature encoder can also be trained end-to-end together with the image generation model.

[0138] The embodiments of the present application do not limit the specific manner in which the electronic device extracts image feature information of the training image.

[0139] In some embodiments, the electronic device performs feature extraction processing on the training image through an image feature encoder to obtain image feature information of the training image. Exemplarily, the image feature encoder includes one or more convolutional layers.

[0140] In some embodiments, in order to improve the image processing speed and computing workload, the electronic device can encode and compress the training image to obtain the image feature information of the training image. For example, the training image is encoded and compressed by the encoder of the variational autoencoder (VAE) or the image feature encoding and decoding in the CLIP (Contrastive Language-Image Pre-Training) model to obtain the image feature information of the training image.

[0141] For example, Figure 4 As shown, the electronic device uses a variational autoencoder (VAE) to encode and compress the training image to obtain the image feature information of the training image. Specifically, the VAE encoder encodes the training image and converts the training image from the pixel space to the latent space to obtain the image feature information of the training image.

[0142] For example, the training image is a photo image of 512*512 pixels, and the data is expressed as (3,512,512), that is, the image data scale of RGB three channels*512*512 pixels, and the data volume is 786432. After VAE encoding, it is converted to the latent space, for example, the image feature information is compressed into (4,64,64) in the latent space, and the data volume is 16384, and the data volume is reduced to 1 / 64 of the original. This can greatly improve the processing efficiency and universality of the image processing method provided in the embodiment of the present application.

[0143] The embodiment of the present application does not limit the specific method for the electronic device to extract text feature information of the prompt word.

[0144] In some embodiments, the electronic device extracts text features of the prompt word through a text encoder to obtain text feature information of the prompt word. Exemplarily, the text encoder includes an embedding layer.

[0145] In some embodiments, Figure 5As shown, the electronic device extracts features of the prompt word through the text encoder in CLIP (Contrastive Language-Image Pre-training) to obtain the text feature information of the prompt word. Among them, the CLIP text encoder converts the text information of the prompt word into information expressed in numbers so that the machine can understand the semantics of the prompt word text. Exemplarily, the CLIP text encoder can be understood as a special Transformer language model, whose input is text and output is a matrix of digital expression, that is, each token in the prompt word is expressed in an Embedding manner, and each token corresponds to a set of Embedding vectors. A series of tokens are combined together to form an Embedding matrix, and the Embedding matrix is ​​recorded as the text feature information of the prompt word.

[0146] In addition to using the above method to extract the image feature information of the training image and the text feature information of the prompt word, the electronic device can also use other methods to obtain them, and the embodiments of the present application are not limited to this.

[0147] The electronic device extracts the image feature information and texture feature information of the training image, and extracts the text feature information of the prompt word, and then performs the following step S104.

[0148] S103: Determine target reference feature information.

[0149] The target reference feature information is used to represent the target live attack feature, and the target live attack belongs to the target live attack type.

[0150] It should be noted that there is no particular order in which S102 and S103 are executed, that is, S102 may be executed before S103 or after S103, or may be executed synchronously with S103.

[0151] The focus of the embodiments of the present application is how to train an image generation model that generates images more finely. As can be seen from the above, the image generated by relying solely on text (i.e., prompt words) as conditions to guide the generation cannot achieve the expected effect, so it is necessary to use the method of the embodiments of the present application to improve the generation effect. First of all, the embodiments of the present application need to find a suitable conditional graph for the training image. In the embodiments of the present application, the image features of the target living attack are used as auxiliary conditions. For example, there is a huge material difference between the mask texture and the real person texture, so the texture is a feature that meets the requirements of living body detection. Based on this, in the embodiments of the present application, when using the training image to train the image generation model, in order to improve the generation accuracy of the living attack image, the target reference feature information is determined, and the target reference feature information is used to represent the image features of the target living attack.

[0152] The embodiment of the present application does not limit the specific method of determining the target reference feature information. Exemplarily, the target reference feature information may be determined in the following ways:

[0153] Method 1: determine information of N candidate reference features corresponding to the target live attack type, and select one candidate reference feature from the information of the N candidate reference features as the target reference feature information, where N is a positive integer.

[0154] The live attack types in the embodiment of the present application include 3D mask, paper copy, screen copy, 3D head model and other attack types. In the embodiment of the present application, different live attack types may include different live attack features.

[0155] For example, 3D masks and 3D head models have obvious differences in material compared to real living bodies, and different materials have different texture features. Therefore, the living body attack features of 3D masks and 3D head models can include features such as texture.

[0156] For another example, compared with real living things, paper copies and screen copies may have features such as shadows and image edges. Therefore, the live attack features of paper copies and screen copies may include features such as shadows and image edges.

[0157] In the first method, the electronic device determines the information of N candidate reference features corresponding to the target live attack type based on the above method. The electronic device selects one candidate reference feature from the information of the N candidate reference features as the target reference feature information. For example, the electronic device determines the information of a default candidate reference feature as the target reference feature information. For another example, the electronic device can display the information of the N candidate reference features to the object, and the object can select one candidate reference feature from the information of the N candidate reference features as the target reference feature information.

[0158] For example, assuming that the type of live attack is a 3D mask, the 3D mask may include different materials, for example, N different materials, and the electronic device determines the N different materials as the information of the N candidate reference features corresponding to the 3D mask. The electronic device displays the N different materials to the object, and the object selects the silicone material among the N different materials as the target reference feature. Based on the object's selection operation of the silicone material, the electronic device extracts the information of the silicone material as the target reference feature information.

[0159] Method 2: If the training image is a live attack image, live attack feature information of the training image is extracted as target reference feature information.

[0160] In the second method, if the training image is a live body attack image, the electronic device can directly determine the live body attack information of the training image as the target reference feature information.

[0161] For example, when the training image is a screen copy, the electronic device extracts edge feature information of the training image as target reference feature information.

[0162] For example, when the training image is a 3D mask or a 3D head model, the electronic device extracts texture feature information of the training image as target reference feature information.

[0163] The embodiment of the present application does not limit the specific method of extracting texture feature information of the training image by the electronic device. For example, the texture feature information of the training image is extracted by gray level co-occurrence matrix, Tamura texture feature, autoregressive texture model, wavelet transform, etc.

[0164] In the second method, the live attack image is used as the training image, and the live attack feature information of the training image is extracted as the target reference feature information, which can obtain accurate target live attack image information, improve the training accuracy of the model, and reduce the difficulty of obtaining the target reference feature information.

[0165] Method three: obtaining a fourth liveness attack image, and extracting liveness attack feature information of the fourth liveness attack image as target reference feature information.

[0166] In this method three, when generating a model based on the training image in the training image, a fourth live attack image is also used. The fourth live attack image can be input by the object, or downloaded by the electronic device from the network. The embodiment of the present application does not limit the specific method of obtaining the fourth live attack image. The electronic device can extract the live attack feature information of the fourth live attack image as the target reference feature information. It should be noted that if the above-mentioned training image is a live attack image, the live attack type of the fourth live attack image can be the same as the live attack type of the training image, or it can be different. The embodiment of the present application does not limit this. Among them, the specific method of extracting the live attack feature information of the fourth live attack image can refer to the relevant method of extracting the live attack feature information of the training image in the above-mentioned method two, and the embodiment of the present application will not repeat it.

[0167] In the third method, by using the live attack feature information of the fourth live attack image as the target reference feature information, accurate target live attack image information can be obtained, thereby improving the training accuracy of the model, and reducing the difficulty of obtaining the target reference feature information, thereby enriching the types of target live attack image information and improving the generalization ability of the model.

[0168] In some embodiments, the information of the N candidate reference features or the live attack feature information of the fourth live attack image is image texture feature information.

[0169] S104: Using the text feature information of the prompt word and the target reference feature information as control conditions, the image feature information of the training image is processed by the image generation model to obtain a third live attack image for the biometric feature image.

[0170] In the embodiment of the present application, in order to make the image generated by the image generation model not only meet the requirements of the prompt word, but also have the image detail features corresponding to the target attack type, when training the image generation model, firstly, the image feature information of the training image is extracted, the text feature information of the prompt word is extracted, and the target reference feature information is determined. Figure 6 As shown, the image feature information of the training image, the text feature information of the prompt word, and the target reference feature information are input into the image generation model, so that the image generation model processes the image feature information of the training image with the text feature information of the prompt word and the target reference feature information as control conditions to generate the third live attack image. Since the target reference feature information and the text feature information of the prompt word are considered when generating the third live attack image, the generated third live attack image not only meets the requirements of the target live attack type indicated by the prompt word, but also includes the image features of the target live attack under the target live attack type, thereby improving the effect of the third live attack image generated by the model.

[0171] The embodiment of the present application does not limit the specific network structure of the above-mentioned image generation model.

[0172] In some embodiments, Figure 7 As shown, the image generation model includes a denoising network, and the above S104 includes the following steps S104-A to S104-C:

[0173] S104-A, performing noise processing on the image feature information of the training image to obtain the noise feature information of the training image;

[0174] S104-B, using the text feature information of the prompt word and the target reference feature information as control conditions, denoising the noisy feature information of the training image through a denoising network to obtain denoised image feature information;

[0175] S104-C, decoding the denoised image feature information to obtain a third living body attack image.

[0176] In this implementation, forward diffusion and reverse diffusion techniques are used to generate the third live attack image. The forward diffusion process adds noise to the image, gradually converting it into a featureless noise image. For example, the forward process will turn any cat or dog image into a noise image. In the end, it will be impossible to tell whether they were originally dogs or cats. Reverse diffusion is the process of subtracting this estimated noise from the noisy image, and eventually an image of a cat or a dog will be obtained.

[0177] In an embodiment of the present application, after the electronic device obtains the image feature information and target reference feature information of the training image and the text feature information of the prompt word based on the above steps, it performs noise processing on the image feature information of the training image, for example, adding a preset noise value to the image feature information of the training image to obtain the noisy feature information of the training image. Exemplarily, Gaussian noise can be gradually added to the image feature information of the training image to obtain the noisy feature information of the training image. This process can be understood as a forward diffusion process.

[0178] Next, the electronic device performs denoising based on the noise feature information of the training image, the target reference feature information, and the text feature information of the prompt word to obtain denoised image feature information. This process can be understood as a reverse diffusion process.

[0179] The embodiment of the present application performs denoising processing on the electronic device based on the noise feature information of the training image, the target reference feature information, and the text feature information of the prompt word, and the specific process of obtaining the denoised image feature information is not limited.

[0180] In some embodiments, the electronic device uses mathematical techniques to perform multiple denoising processes on the noisy feature information of the training image with the text feature information of the prompt word and the target reference feature information as control conditions to obtain denoised image feature information.

[0181] In some embodiments, the above S104-B includes the following steps S104-B1 to S104-B3:

[0182] S104-B1, using the text feature information of the prompt word and the target reference feature information as control conditions, denoising the i-th denoised feature information of the training image through the denoising network to obtain the i-th noise prediction value, where i is a positive integer less than or equal to N, and if i is 1, the i-th denoised feature information is the noisy feature information of the training image;

[0183] S104-B2, obtaining the (i+1)th denoising feature information of the training image based on the i-th noise prediction value and the i-th denoising feature information of the training image;

[0184] S104-B3, taking the (i+1)th denoising feature information as the new (i)th denoising feature information, returning to repeat the above steps to obtain the (N)th denoising feature information of the training image;

[0185] S104-B4, determining the Nth denoising feature information as the denoised image feature information.

[0186] In this implementation, the electronic device uses a denoising network to perform multiple denoising processes on the noisy feature information of the training image with the text feature information of the prompt word and the target reference feature information as control conditions. The denoising network is used to predict the noise value of each denoising process. Specifically, during each denoising, the noise value currently predicted by the denoising network is subtracted from the previous denoising feature information to generate new denoising feature information and input it into the denoising network for further denoising. Repeat multiple times until the noise added to the training image is removed. In each denoising process, in addition to inputting the new denoising feature information of the training image and the text feature information of the prompt word, the denoising network also inputs the target reference feature information, so that the image feature information is close to the text feature of the prompt word and the target reference feature information during the denoising process, so that the text feature of the prompt word and the target reference feature information are integrated into the image feature information obtained by the final denoising, so that the third live attack image generated based on the image feature information obtained by denoising not only meets the instructions of the prompt word, but also includes the target live attack feature, thereby improving the image generation effect of the image generation model.

[0187] The following introduces the specific process of using a denoising network to denoise the noisy feature information of the training image with the text feature information of the prompt word and the target reference feature information as control conditions to obtain the denoised image feature information.

[0188] like Figure 8 As shown, the denoising process using the denoising network is an iterative process. For example, the denoising network is used to denoise the i-th denoised feature information of the training image with at least one of the text feature information of the prompt word and the target reference feature information as a control condition to obtain the i-th noise prediction value. Then, based on the i-th noise prediction value and the i-th denoised feature information of the training image, the i+1-th denoised feature information of the training image is obtained. Then, the i+1-th denoised feature information is used as the new i-th denoised feature information, and the denoising network is used to denoise the new i-th denoised feature information with at least one of the text feature information of the prompt word and the target reference feature information as a control condition to obtain the i-th noise prediction value. Based on the i-th noise prediction value and the i-th denoised feature information of the training image, the i+1-th denoised feature information of the training image is obtained. Next, the i+1th denoising feature information is used as the new i-th denoising feature information and returned for execution. After repeating the denoising operation N times, the N-th denoising feature information can be obtained, and then the N-th denoising feature information is determined as the denoised image feature information.

[0189] For example, suppose N is equal to 20, such as Fig. 9As shown, first, the electronic device inputs the noise feature information of the training image, the text feature information of the prompt word and / or the target reference feature information into the denoising network for denoising, and obtains the first noise prediction value preset by the denoising network. Based on the first noise prediction value and the noise feature information of the training image, the first denoising feature information of the training image is obtained. Next, the first denoising feature information of the training image, the text feature information of the prompt word and / or the target reference feature information are input into the denoising network for denoising, and the second noise prediction value preset by the denoising network is obtained. Based on the second noise prediction value and the first denoising feature information, the second denoising feature information is obtained. Next, the second denoising feature information of the training image, the text feature information of the prompt word and / or the target reference feature information are input into the denoising network for denoising, and the third noise prediction value preset by the denoising network is obtained. Based on the third noise prediction value and the second denoising feature information, the third denoising feature information is obtained. Repeat the process, input the 19th denoising feature information of the training image, the text feature information of the prompt word and / or the target reference feature information into the denoising network for denoising, obtain the 20th noise prediction value preset by the denoising network, and obtain the 20th denoising feature information based on the 20th noise prediction value and the 19th denoising feature information. Then, determine the 20th denoising feature information as the denoised image feature information.

[0190] The following is an introduction to the specific process of using the text feature information of the prompt word and the target reference feature information as control conditions in the above S104-B1, performing denoising on the i-th denoising feature information of the training image through the denoising network, and obtaining the i-th noise prediction value.

[0191] In an embodiment of the present application, the i-th denoising feature information of the training image is denoised by a denoising network, and at least one of the text feature information of the prompt word and the target reference feature information can be fused with the output feature information of at least part of the network layers in the denoising network, thereby achieving the binding of the text feature information of the prompt word and the target reference feature information as conditions and the distribution of the training image.

[0192] In the embodiment of the present application, the electronic device uses the text feature information of the prompt word and the target reference feature information as control conditions, and performs denoising on the i-th denoising feature information of the training image through a denoising network, and the specific method of obtaining the i-th noise prediction value is not limited.

[0193] In some embodiments, the above Fig.10 As shown, the denoising network includes multiple network layers (for example, M network layers), and the above S104-B1 includes the following steps S104-B11 and S104-B12:

[0194] S104-B11, using the i-th denoising feature information of the training image as input feature information of the denoising network;

[0195] S104-B12, the feature information output by some network layers of the denoising network is at least fused with the text feature information of the prompt word, and the feature information output by some network layers of the denoising network is at least fused with the target reference feature information to obtain the i-th noise prediction value.

[0196] In the i-th denoising process, the electronic device obtains the i-th denoising feature information of the training image obtained in the i-1th denoising process, and inputs the i-th denoising feature information of the training image into the denoising network. In some embodiments, when the i-th denoising feature information of the training image is used as the input feature information of the denoising network in S104-B11, the electronic device may also fuse the i-th denoising feature information of the training image with at least one of the text feature information of the prompt word and the target reference feature information, and use the fused feature information as the input feature information of the first network layer of the denoising network.

[0197] like Fig.10 As shown, the denoising network includes multiple network layers. The electronic device fuses the feature information output by some network layers of the denoising network with at least the text feature information of the prompt word, and fuses the feature information output by some network layers of the denoising network with at least the target reference feature information to obtain the i-th noise prediction value.

[0198] In one example, if Fig.11 As shown, the electronic device fuses the feature information output by each network layer of the denoising network with the text feature information of the prompt word and the target reference feature information, and uses the fused feature information as the input feature information of the next network layer. Similarly, the feature information output by the last network layer of the denoising network is obtained as the i-th noise prediction value.

[0199] In one example, the electronic device fuses the output feature information of some network layers in the denoising network with the text feature information of the prompt word, and fuses the output feature information of some network layers in the denoising network with the target reference feature information. Fig.12 As shown, the electronic device fuses the feature information 1 output by the first network layer of the denoising network with the text feature information of the prompt word, and uses the fused feature information as the input feature information of the second network layer. After fusing the output feature information 2 of the second network layer with the target reference feature information, it is used as the input feature information of the third network layer, and so on, to obtain the feature information output by the last network layer of the denoising network as the i-th noise prediction value.

[0200] In one example, the electronic device fuses the output feature information of some network layers in the denoising network with at least one of the text feature information of the prompt word and the target reference feature information, and fuses the output feature information of some network layers in the denoising network with the text feature information of the prompt word and the target reference feature information. Fig.13 As shown, the electronic device fuses the feature information 1 output by the first network layer of the denoising network with the text feature information of the prompt word, and uses the fused feature information as the input feature information of the second network layer. The output feature information 2 of the second network layer is fused with the target reference feature information and used as the input feature information of the third network layer. The output feature information 3 of the third network layer is fused with the target reference feature information and the text feature information of the prompt word and used as the input feature information of the fourth network layer. And so on, the feature information output by the last network layer of the denoising network is obtained as the i-th noise prediction value.

[0201] The embodiment of the present application does not limit the specific manner of fusing the output feature information of the network layer in the denoising network with at least one of the text feature information of the prompt word and the target reference feature information. It is only necessary to ensure that the denoising network considers the target reference feature information and the text feature information of the prompt word when denoising the i-th denoising feature information of the training image.

[0202] In the embodiment of the present application, in order to facilitate the description of the process of fusing the output feature information of the network layer in the denoising network with the text feature information of the prompt word and at least one of the target reference feature information, the kth network layer in the denoising network is used as an example for explanation. At this time, the above S104-B12 may include the following steps S104-B12-a and S104-B12-b:

[0203] S104-B12-a, for the kth network layer in the denoising network, fusing the output feature information of the kth network layer with at least one of the text feature information of the prompt word and the target reference feature information to obtain the input feature information of the k+1th network layer in the denoising network, where k is a positive integer;

[0204] S104-B12-b. Based on the input feature information of the k+1th network layer, obtain the i-th noise prediction value.

[0205] The embodiments of the present application do not limit the specific network structure of the denoising network. For example, it can be a neural network that predicts the noise value based on any noisy image feature information and text feature information. For example, a U-Net network can be used to diffuse the denoised image feature information in the direction indicated by the prompt word while removing the noise in the image information, so as to generate image feature information including the prompt information.

[0206] In the embodiment of the present application, for the kth network layer of the denoising network, the electronic device fuses the output feature information of the kth network layer with at least one of the text feature information of the prompt word and the target reference feature information to obtain the input feature information of the k+1th network layer in the denoising network. For example, Fig.14A As shown, the electronic device fuses the output feature information of the kth network layer with the text feature information of the prompt word, and uses the fused feature information as the input feature information of the k+1th network layer in the denoising network. Fig. 14B As shown, the electronic device fuses the output feature information of the kth network layer with the target reference feature information, and uses the fused feature information as the input feature information of the k+1th network layer in the denoising network. Fig. 14C As shown, the electronic device fuses the output feature information of the kth network layer with the target reference feature information and the text feature information of the prompt word, and uses the fused feature information as the input feature information of the k+1th network layer in the denoising network.

[0207] The embodiment of the present application does not limit the specific manner in which the electronic device fuses the output feature information of the kth network layer with at least one of the text feature information of the prompt word and the target reference feature information to obtain the input feature information of the k+1th network layer in the denoising network.

[0208] In some embodiments, the electronic device fuses the output feature information of the kth network layer with the text feature information of the prompt word and at least one of the target reference feature information through one or more convolutional layers to obtain the input feature information of the k+1th network layer in the denoising network.

[0209] The embodiment of the present application does not limit the specific manner in which the output feature information of the kth network layer is fused with the text feature information of the prompt word and at least one of the target reference feature information. For example, the output feature information of the kth network layer is added or multiplied, or concatenated with the text feature information of the prompt word and at least one of the target reference feature information to obtain the input feature information of the k+1th network layer. In some embodiments, the fused feature information may be subjected to other processing (such as convolution or other processing) as the input feature information of the k+1th network layer. In some embodiments, convolution or other processing may be performed before or after the output feature information of the kth network layer, the text feature information of the prompt word, and at least one of the target reference feature information, and then fused, and the embodiment of the present application does not limit this.

[0210] In some embodiments, the electronic device fuses the output feature information of the kth network layer with at least one of the text feature information of the prompt word and the target reference feature information through an attention mechanism to obtain the input feature information of the k+1th network layer.

[0211] For example, the text feature information of the prompt word is used as the key value of the attention module, the target reference feature information is used as the representative value of the attention module, and the output feature information of the kth network layer is used as the query value of the attention module. They are input into the attention module for fusion processing to obtain the input feature information of the k+1th network layer.

[0212] In a possible implementation, the electronic device may perform information fusion according to the following formula (1):

[0213] (1)

[0214] in, is the fused feature information, is the output feature information of the kth network layer in the denoising network, is the text feature information of the prompt word, V is the target reference feature information, and d is the scaling factor, which is used to adjust the input scale of the softmax function.

[0215] For another example, the text feature information of the prompt word is used as the representative value of the attention module, the target reference feature information is used as the key value of the attention module, and the output feature information of the kth network layer is used as the query value of the attention module. They are input into the attention module for fusion processing to obtain the input feature information of the k+1th network layer.

[0216] The above is an introduction to the specific process of an electronic device fusing the output feature information of the kth network layer in the denoising network with at least one of the target reference feature information and the text feature information of the prompt word as the input feature information of the k+1th network layer. Based on the above steps, the electronic device can determine the feature information output by the last network layer of the denoising network.

[0217] For example, Fig.15As shown, assuming that the denoising network includes 4 network layers and 3 attention modules, in the i-th denoising process, the electronic device inputs the i-th denoising feature information of the training image into the first network layer for processing, and obtains the output feature information 1 of the first network layer. Then, the output feature information 1 of the first network layer is input into the first attention module with the text feature information of the prompt word and the target reference feature information for fusion processing, and the fusion feature information 1 output by the first attention module is obtained. The fusion feature information 1 is input into the second network layer of the denoising network for processing, and the output feature information 2 of the second network layer is obtained. Then, the output feature information 2 of the second network layer is input into the second attention module with the text feature information of the prompt word and the target reference feature information for fusion processing, and the fusion feature information 2 output by the second attention module is obtained. The fusion feature information 2 is input into the third network layer of the denoising network for processing, and the output feature information 3 of the third network layer is obtained. Then, the output feature information 3 of the third network layer is input into the third attention module with the text feature information of the prompt word and the target reference feature information for fusion processing, and the fusion feature information 4 output by the third attention module is obtained. The fused feature information 4 is input into the last network layer of the denoising network for processing to obtain the output feature information 4 of the last network layer. Then, based on the output feature information 4 of the last network layer, the i-th noise prediction value is obtained.

[0218] In one example, the electronic device may determine the feature information output by the last network layer of the denoising network as the i-th noise prediction value.

[0219] Next, the above step S104 - B2 is executed to obtain the (i+1)th denoising feature information of the training image based on the i-th noise prediction value and the i-th denoising feature information of the training image.

[0220] For example, the difference between the i-th denoising feature information of the training image and the i-th noise prediction value is determined as the i+1-th denoising feature information of the training image.

[0221] The electronic device uses the i+1th denoising feature information of the training image determined above as the new i-th denoising feature information, returns to execute the above S104-B1 step, repeats it N times, obtains the N-th denoising feature information of the training image, and then uses the N-th denoising feature information as the denoised image feature information.

[0222] Finally, the electronic device executes the above step S104-C, that is, the electronic device decodes the denoised image feature information to obtain a third live attack image.

[0223] In an embodiment of the present application, the above-mentioned denoised image feature information is feature information in the latent space, and the electronic device needs to decode the denoised image feature information to convert the denoised image feature information from the latent space to the pixel space to obtain the third living attack image.

[0224] The embodiment of the present application does not limit the specific manner in which the electronic device decodes the denoised image feature information to obtain the third live attack image. For example, the electronic device decodes the denoised image feature information through an image decoder to obtain the third live attack image.

[0225] In some embodiments, Fig.16 As shown, the electronic device decodes the denoised image feature information through the VAE decoder to obtain a third live attack image. The third live attack image transforms the initial features of the object in the training image into the target features described by the prompt word based on the prompt word and the target reference feature information, thereby improving the image generation effect.

[0226] It should be noted that the VAE decoder can be a pre-trained decoder. Optionally, the VAE decoder can be trained end-to-end together with the image generation model.

[0227] Based on the above steps, the electronic device processes the image feature information of the training image, the text feature information of the prompt word and the target reference feature information through the image generation model to generate a third live attack image. Next, the following step S105 is executed.

[0228] S105: Determine the loss of the image generation model based on the target reference feature information and the third living attack image, and train the image generation model based on the loss.

[0229] In the embodiment of the present application, the electronic device image generation model processes the image feature information of the training image, the text feature information of the prompt word, and the target reference feature information to generate a third live attack image. Then, based on the third live attack image and the target reference feature information, the loss of the image generation model is determined. Then, based on the loss, the image generation model is trained.

[0230] The embodiment of the present application does not limit the specific manner in which the electronic device determines the loss of the image generation model based on the third living attack image and the target reference feature information.

[0231] In the embodiment of the present application, it is hoped that the live attack feature information of the third live attack image restored by the image generation model is consistent with the target reference feature information. Based on this, the above S105 includes the following steps S105-A and S105-B:

[0232] S105-A, extracting liveness attack features from the third liveness attack image to obtain liveness attack feature information corresponding to the third liveness attack image;

[0233] S105-B: Determine the loss of the image generation model based on the target reference feature information and the live attack feature information of the third live attack image.

[0234] In this implementation, when the electronic device generates the loss of the image generation model based on the third live attack image and the target reference feature information, the electronic device extracts the live attack feature of the third live attack image to obtain the live attack feature information corresponding to the third live attack image. Then, based on the live attack feature information of the third live attack image and the target reference feature information, the electronic device determines the loss of the image generation model.

[0235] The embodiment of the present application does not limit the specific manner in which the electronic device determines the loss of the image generation model based on the live attack feature information and the target reference feature information of the third live attack image. For example, the live attack feature information and the target reference feature information corresponding to the third live attack image are brought into a preset loss function to calculate the loss of the image generation model. The embodiment of the present application does not limit the specific type of the preset loss function, for example, it may be a cross entropy loss function. For another example, the difference between the live attack feature information and the target reference feature information corresponding to the third live attack image is determined, and the loss of the model is determined based on the difference.

[0236] In some embodiments, if the target reference feature information is the texture feature information of the training image, then the extracting the liveness attack feature of the third liveness attack image to obtain the liveness attack feature information of the third liveness attack image includes: extracting the texture feature of the third liveness attack image to obtain the texture feature information of the third liveness attack image. Correspondingly, determining the loss of the image generation model based on the target reference feature information and the liveness attack feature information of the third liveness attack image includes: determining the loss of the image generation model based on the texture feature information of the training image and the texture feature information of the third liveness attack image.

[0237] The embodiment of the present application does not limit the specific manner in which the electronic device determines the loss of the image generation model based on the texture feature information of the third live attack image and the texture feature information of the training image. For example, the texture feature information of the third live attack image and the texture feature information of the training image are brought into a preset loss function to calculate the loss of the image generation model. The embodiment of the present application does not limit the specific type of the preset loss function, for example, it can be a cross entropy loss function, etc.

[0238] In some embodiments, the electronic device determines a difference between texture feature information of the training image and texture feature information of the third living attack image; and determines a loss of the image generation model based on the difference.

[0239] In one example, the loss of the image generation model includes a texture loss.

[0240] For example, the electronic device may use the following formula (2) to determine the texture loss: :

[0241] (2)

[0242] Among them, f ori represents the texture feature information of the training image, and f rec Represents the texture feature information of the third living body attack image. is the Euclidean distance.

[0243] In one example, the loss of the image generation model in the embodiment of the present application also includes diffusion loss. The diffusion loss can be calculated using the MSE loss function. The embodiment of the present application does not limit this.

[0244] After the electronic device determines the loss of the image generation model based on the above steps, it updates the parameters in the image generation model based on the loss. At the same time, it determines whether the training end condition of the image generation model is met, where the training end condition may include that the loss meets the preset loss, or the number of training times reaches the preset number of times. If the model training end condition is not met at present, a new batch of training samples is selected, and the image generation model with updated parameters is trained again using this batch of training samples in the same way as above, and the execution is repeated until the model training end condition is met, and finally the trained image generation model is obtained.

[0245] The model training method provided in the embodiment of the present application obtains a training image and a prompt word, the prompt word is used to indicate the target live attack type, and the training image is a biometric image or a live attack image; then, image feature extraction and texture feature extraction are performed on the training image to obtain image feature information of the training image, and text feature extraction is performed on the prompt word to obtain text feature information of the prompt word, and target reference feature information is determined, the target reference feature information is used to represent the image features of the target live attack, and the target live attack belongs to the target live attack type. Then, the text feature information of the prompt word and the target reference feature information are used as control conditions, and the image feature information of the training image is processed by the image generation model to obtain a third live attack image for the biometric image, and based on the training image and the third live attack image, the loss of the image generation model is determined, and the image generation model is trained based on the loss. It can be seen that when the embodiment of the present application generates the third live attack image based on the training image and the prompt word, the target reference feature information and the prompt word are used as control conditions to process the training image to generate the third live attack image. The generated third live attack image not only meets the type of target live attack indicated by the prompt word, but also includes the image detail features of the target live attack under the target live attack type, thereby improving the generation quality of the third live attack image generated by the model. In addition, in the embodiment of the present application, based on the target reference feature information and the third live attack image, the loss of the image generation model is determined, so that the image generation model can pay attention to the image detail features of the live attack, thereby improving the image generation performance of the image generation model.

[0246] The above introduces the training process of the image generation model. The following introduces the image processing process based on the trained image generation model.

[0247] Fig.17 The flowchart of the image processing method provided by an embodiment of the present application is shown in FIG. The execution subject of the embodiment of the present application may be a device having an image processing function, such as an image processing device. The image processing device may be the above-mentioned Figure 1 The server 102 or terminal device 101 shown may also be Figure 1 The system is composed of the server 102 or the terminal device 101. For the convenience of description, the following embodiments take electronic devices as an example to illustrate the method of the embodiments of the present application.

[0248] like Fig.17 As shown, the image processing method of the embodiment of the present application includes:

[0249] S201: Acquire a first image and a prompt word.

[0250] The prompt word is used to indicate the target live attack type, and the first image is a biometric feature image.

[0251] The image processing method of the embodiment of the present application can be applied in different fields to generate images that meet different requirements.

[0252] In one example, the image processing method of the embodiment of the present application can be applied to the field of AI drawing to generate images that meet the requirements.

[0253] In one example, the image processing method of the embodiment of the present application can be applied to the field of generating training samples to generate training samples that meet the requirements. For example, the image processing method can be used to generate training data for a liveness detection model, that is, to generate realistic, high-quality liveness attack samples.

[0254] In the embodiment of the present application, when the image processing method is used in different fields, a first image of the corresponding field can be acquired.

[0255] The first image in the embodiment of the present application can be understood as an image to be processed or modified. The first image includes at least one object.

[0256] In the embodiment of the present application, the prompt word is used to indicate the type of attack on the target living body.

[0257] The types of live attack in the embodiment of the present application include 3D mask, paper copy, screen copy, 3D head model and other attack types. In the embodiment of the present application, different live attack types may include different live attack features. In the embodiment of the present application, the target live attack type may be indicated by a prompt word, for example, indicating that the target live attack type is 3D mask or screen copy, etc.

[0258] The embodiment of the present application does not limit the specific manner in which the electronic device obtains the first image and the prompt word.

[0259] In a possible implementation, the prompt word and the first image are both input by the object.

[0260] In a possible implementation, the first image is generated by an electronic device, and the prompt word is input by the subject.

[0261] In some embodiments, the first image is a live attack image. In some embodiments, the first image is a real live image. That is, the embodiment of the present application can generate another live attack image based on a live attack image, or generate a live attack image based on a real live image.

[0262] S202: Extract image feature information of the first image and text feature information of the prompt word.

[0263] In an embodiment of the present application, after the electronic device obtains the first image and the prompt word corresponding to the first image based on the above steps, in order to achieve mutual fusion between the image information of the first image and the semantic information of the prompt word, the electronic device first extracts the image feature information of the first image and the text feature information of the prompt word.

[0264] In some embodiments, before extracting image feature information of the first image, the electronic device first preprocesses the first image to obtain a preprocessed first image.

[0265] The specific manner of preprocessing the first image in the embodiment of the present application is not limited, for example, the clarity and sharpness of the first image are processed.

[0266] In one example, the electronic device crops the size of the first image to be consistent with the size of the input image of the image generation model based on the size of the input image of the image generation model.

[0267] In some examples, the electronic device performs a cutout process on the first image to cut out an object region in the first image, and then adjusts the size of the object region to a first size. Then, the electronic device performs a center cropping of the object region of the first size to a second size to obtain a preprocessed first image.

[0268] Next, the electronic device extracts image features from the preprocessed first image.

[0269] like Fig.18 As shown, the embodiment of the present application encodes the first image through an image feature encoder to compress the first image from the pixel space to the latent space to obtain the image feature information of the first image. The main purpose of this part is to represent the image using high-level semantic features, which can not only reduce the dimension of the original image space to improve the image speed, but also facilitate better interaction with subsequent conditional features (i.e., the text feature information of the prompt word).

[0270] The embodiment of the present application does not limit the specific network structure of the image feature encoding and decoding. Exemplarily, the image feature encoding and decoding can be the image feature encoding and decoding information in the VAE encoder or CLIP.

[0271] In the embodiment of the present application, the electronic device not only extracts the image feature information of the first image, but also extracts the text feature information of the prompt word.

[0272] like Fig.18 As shown, the embodiment of the present application uses a text encoder to extract text features of the prompt word in the electronic device to obtain text feature information of the prompt word. The embodiment of the present application does not limit the specific network structure of the text encoder. For example, the text encoder can be a CLIP text encoder.

[0273] In addition to using the above method to extract the image feature information of the first image and the text feature information of the prompt word, the electronic device can also use other methods to obtain them, and the embodiments of the present application do not limit this.

[0274] The electronic device extracts the image feature information of the first image and the text feature information of the prompt word, and executes the following step S203.

[0275] S203: Determine target reference feature information.

[0276] The target reference feature information is used to represent the target live attack feature, and the target live attack belongs to the target live attack type.

[0277] It should be noted that there is no particular order of execution between S202 and S203 , that is, S202 may be executed before S203 , or after S203 , or may be executed synchronously with S203 .

[0278] The embodiment of the present application does not limit the specific method of determining the target reference feature information. Exemplarily, the target reference feature information may be determined in the following ways:

[0279] Method 1: determine information of N candidate reference features corresponding to the target live attack type, and select one candidate reference feature from the information of the N candidate reference features as the target reference feature information, where N is a positive integer.

[0280] The live attack types in the embodiment of the present application include 3D mask, paper copy, screen copy, 3D head model and other attack types. In the embodiment of the present application, different live attack types may include different live attack features.

[0281] For example, 3D masks and 3D head models have obvious differences in material compared to real living bodies, and different materials have different texture features. Therefore, the living body attack features of 3D masks and 3D head models can include features such as texture.

[0282] For another example, compared with real living things, paper copies and screen copies may have features such as shadows and image edges. Therefore, the live attack features of paper copies and screen copies may include features such as shadows and image edges.

[0283] In the first method, the electronic device determines the information of N candidate reference features corresponding to the target live attack type based on the above method. The electronic device selects one candidate reference feature from the information of the N candidate reference features as the target reference feature information. For example, the electronic device determines the information of a default candidate reference feature as the target reference feature information. For another example, the electronic device can display the information of the N candidate reference features to the object, and the object can select one candidate reference feature from the information of the N candidate reference features as the target reference feature information.

[0284] Method three: obtaining a second liveness attack image, extracting liveness attack feature information of the second liveness attack image as target reference feature information.

[0285] For example, when the second liveness attack image is a screen copy, the electronic device extracts edge feature information of the second liveness attack image as target reference feature information.

[0286] For example, when the second live attack image is a 3D mask or a 3D head model, the electronic device extracts texture feature information of the second live attack image as target reference feature information.

[0287] The embodiment of the present application does not limit the specific manner in which the electronic device extracts the texture feature information of the second live attack image. For example, Fig.18 As shown, the electronic device extracts texture feature information of the second live attack image through a texture feature extraction module. Optionally, the texture feature extraction module may include a gray level co-occurrence matrix, a Tamura texture feature, an autoregressive texture model, a wavelet transform, and the like.

[0288] In some embodiments, the information of the N candidate reference features or the live attack feature information of the second live attack image is image texture feature information.

[0289] S204: Using the text feature information of the prompt word and the target reference feature information as control conditions, the image feature information of the first image is processed by the image generation model to obtain a first live attack image for the biometric feature image.

[0290] In the embodiment of the present application, in order to make the image generated by the image generation model not only meet the requirements of the prompt word, but also have the image detail features corresponding to the target attack type, during image processing, the image feature information of the first image is first extracted, the text feature information of the prompt word is extracted, and the target reference feature information is determined. Fig.18As shown, the image feature information of the first image, the text feature information of the prompt word, and the target reference feature information are input into the image generation model, so that the image generation model processes the image feature information of the first image with the text feature information of the prompt word and the target reference feature information as control conditions to generate the first live attack image. Since the target reference feature information and the text feature information of the prompt word are considered when generating the first live attack image, the generated first live attack image not only meets the requirements of the target live attack type indicated by the prompt word, but also includes the image features of the target live attack under the target live attack type, thereby improving the generation effect of the first live attack image.

[0291] The embodiment of the present application does not limit the specific network structure of the above-mentioned image generation model.

[0292] In some embodiments, if the image generation model includes a denoising network, then the above S204 includes the following steps S204-A to S204-C:

[0293] S204-A, performing noise processing on the image feature information of the first image to obtain noise feature information of the first image;

[0294] S204-B, using the text feature information of the prompt word and the target reference feature information as control conditions, denoising the noisy feature information of the first image through a denoising network to obtain denoised image feature information;

[0295] S204-C, decoding the denoised image feature information to obtain a first living body attack image.

[0296] In this implementation, forward diffusion and reverse diffusion techniques are used to generate the first live attack image. The forward diffusion process adds noise to the image, gradually converting it into a featureless noise image. Reverse diffusion is the process of subtracting the estimated noise from the noisy image.

[0297] In an embodiment of the present application, after the electronic device obtains the image feature information and texture feature information of the first image and the text feature information of the prompt word based on the above steps, it performs noise processing on the image feature information of the first image, for example, adding a preset noise value to the image feature information of the first image to obtain the noisy feature information of the first image. Exemplarily, Gaussian noise can be gradually added to the image feature information of the first image to obtain the noisy feature information of the first image. This process can be understood as a forward diffusion process.

[0298] Next, the electronic device performs denoising processing based on the noise feature information of the first image, the target reference feature information, and the text feature information of the prompt word to obtain denoised image feature information. This process can be understood as a reverse diffusion process.

[0299] In some embodiments, the above S204-B includes the following steps S204-B1 to S204-B3:

[0300] S204-B1, using the text feature information of the prompt word and the target reference feature information as control conditions, denoising the i-th denoised feature information of the first image through a denoising network to obtain an i-th noise prediction value, where i is a positive integer less than or equal to N, and if i is 1, the i-th denoised feature information is the noisy feature information of the first image;

[0301] S204-B2, obtaining (i+1) th denoising feature information of the first image based on the i th noise prediction value and the i th denoising feature information of the first image;

[0302] S204-B3, taking the (i+1)th denoising feature information as new (i)th denoising feature information, returning to repeat the above steps, and obtaining the (N)th denoising feature information of the first image;

[0303] S204-B4, determining the Nth denoising feature information as the denoised image feature information.

[0304] In this implementation, the electronic device uses a denoising network to perform multiple denoising processes on the noisy feature information of the first image, using the text feature information of the prompt word and the target reference feature information as control conditions. The denoising network is used to predict the noise value of each denoising process. Specifically, during each denoising process, the noise value currently predicted by the denoising network is subtracted from the previous denoising feature information to generate new denoising feature information that is input into the denoising network for further denoising. This is repeated multiple times until the noise added to the first image is completely removed. In each denoising process, in addition to the new denoising feature information of the first image and the text feature information of the prompt word, the target reference feature information is also input into the denoising network, so that the image feature information is close to the text feature of the prompt word and the target reference feature information during the denoising process, so that the text feature of the prompt word and the target reference feature information are integrated into the image feature information obtained by denoising. In this way, the first live attack image generated based on the denoising image feature information not only meets the requirements of the target live attack type indicated by the prompt word, but also includes the image detail features of the target live attack under the target live attack type, thereby improving the image generation effect of the image generation model.

[0305] The following introduces a specific process of using a denoising network to denoise the noisy feature information of the first image with the text feature information of the prompt word and the target reference feature information as control conditions to obtain the denoised image feature information.

[0306] In an embodiment of the present application, the denoising process using the denoising network is an iterative process. For example, the denoising network is used to perform denoising on the i-th denoised feature information of the first image with at least one of the text feature information of the prompt word and the target reference feature information as a control condition to obtain the i-th noise prediction value. Then, based on the i-th noise prediction value and the i-th denoised feature information of the first image, the i+1-th denoised feature information of the first image is obtained. Then, the i+1-th denoised feature information is used as the new i-th denoised feature information, and the denoising network is used to perform denoising on the new i-th denoised feature information with at least one of the text feature information of the prompt word and the target reference feature information as a control condition to obtain the i-th noise prediction value. Based on the i-th noise prediction value and the i-th denoised feature information of the first image, the i+1-th denoised feature information of the first image is obtained. Next, the i+1th denoising feature information is used as the new i-th denoising feature information and returned for execution. After repeating the denoising operation N times, the N-th denoising feature information can be obtained, and then the N-th denoising feature information is determined as the denoised image feature information.

[0307] The following is an introduction to the specific process of using the text feature information of the prompt word and the target reference feature information as control conditions in the above S204-B1, performing denoising on the i-th denoising feature information of the first image through a denoising network, and obtaining the i-th noise prediction value.

[0308] In an embodiment of the present application, the i-th denoising feature information of the first image is denoised by a denoising network, and at least one of the text feature information of the prompt word and the target reference feature information can be fused with the output feature information of at least part of the network layers in the denoising network, thereby achieving binding of the text feature information of the prompt word and the target reference feature information as conditions and the distribution of the first image.

[0309] In the embodiment of the present application, the electronic device uses the text feature information of the prompt word and the target reference feature information as control conditions, and performs denoising on the i-th denoising feature information of the first image through a denoising network, and the specific method of obtaining the i-th noise prediction value is not limited.

[0310] In some embodiments, if the denoising network includes multiple network layers, the above S204-B1 includes the following steps S204-B11 and S204-B12:

[0311] S204-B11, using the i-th denoising feature information of the first image as input feature information of the denoising network;

[0312] S204-B12, the feature information output by some network layers of the denoising network is at least fused with the text feature information of the prompt word, and the feature information output by some network layers of the denoising network is at least fused with the target reference feature information to obtain the i-th noise prediction value.

[0313] In one example, the denoising network includes multiple network layers. The electronic device fuses the feature information output by some network layers of the denoising network with at least the text feature information of the prompt word, and fuses the feature information output by some network layers of the denoising network with at least the target reference feature information to obtain the i-th noise prediction value.

[0314] In one example, the electronic device fuses the feature information output by each network layer of the denoising network with the text feature information of the prompt word and the target reference feature information, and uses the fused feature information as the input feature information of the next network layer. By analogy, the feature information output by the last network layer of the denoising network is obtained as the i-th noise prediction value.

[0315] In one example, the electronic device fuses the output feature information of some network layers in the denoising network with the text feature information of the prompt word, and fuses the output feature information of some network layers of the denoising network with the target reference feature information. For example, the electronic device fuses the feature information 1 output by the first network layer of the denoising network with the text feature information of the prompt word, and uses the fused feature information as the input feature information of the second network layer. After fusing the output feature information 2 of the second network layer with the target reference feature information, it is used as the input feature information of the third network layer, and so on, to obtain the feature information output by the last network layer of the denoising network as the i-th noise prediction value.

[0316] In one example, the electronic device fuses the output feature information of some network layers in the denoising network with at least one of the text feature information of the prompt word and the target reference feature information, and fuses the output feature information of some network layers in the denoising network with the text feature information of the prompt word and the target reference feature information. For example, the electronic device fuses the feature information 1 output by the first network layer of the denoising network with the text feature information of the prompt word, and uses the fused feature information as the input feature information of the second network layer. After fusing the output feature information 2 of the second network layer with the target reference feature information, it is used as the input feature information of the third network layer. After fusing the output feature information 3 of the third network layer with the target reference feature information and the text feature information of the prompt word, it is used as the input feature information of the fourth network layer. And so on, the feature information output by the last network layer of the denoising network is obtained as the i-th noise prediction value.

[0317] The embodiment of the present application does not limit the specific manner in which the output feature information of the network layer in the denoising network is integrated with at least one of the text feature information of the prompt word and the target reference feature information. It is only necessary to ensure that the denoising network considers the target reference feature information and the text feature information of the prompt word when denoising the i-th denoising feature information of the first image.

[0318] In the embodiment of the present application, in order to facilitate the description of the process of fusing the output feature information of the network layer in the denoising network with the text feature information of the prompt word and at least one of the target reference feature information, the kth network layer in the denoising network is used as an example for description. At this time, the above S204-B12 may include the following steps S204-B12-a and S204-B12-b:

[0319] S204-B12-a, for the kth network layer in the denoising network, fusing the output feature information of the kth network layer with at least one of the text feature information of the prompt word and the target reference feature information to obtain the input feature information of the k+1th network layer in the denoising network, where k is a positive integer;

[0320] S204-B12-b. Based on the input feature information of the k+1th network layer, obtain the i-th noise prediction value.

[0321] The embodiments of the present application do not limit the specific network structure of the denoising network. For example, it can be a neural network that predicts the noise value based on any noisy image feature information and text feature information. For example, a U-Net network can be used to diffuse the denoised image feature information in the direction indicated by the prompt word while removing the noise in the image information, so as to generate image feature information including the prompt information.

[0322] In an embodiment of the present application, for the kth network layer of the denoising network, the electronic device fuses the output feature information of the kth network layer with at least one of the text feature information of the prompt word and the target reference feature information to obtain the input feature information of the k+1th network layer in the denoising network.

[0323] The embodiment of the present application does not limit the specific manner in which the electronic device fuses the output feature information of the kth network layer with at least one of the text feature information of the prompt word and the target reference feature information to obtain the input feature information of the k+1th network layer in the denoising network.

[0324] In some embodiments, the electronic device fuses the output feature information of the kth network layer with the text feature information of the prompt word and at least one of the target reference feature information through one or more convolutional layers to obtain the input feature information of the k+1th network layer in the denoising network.

[0325] In some embodiments, the electronic device fuses the output feature information of the kth network layer with at least one of the text feature information of the prompt word and the target reference feature information through an attention mechanism to obtain the input feature information of the k+1th network layer.

[0326] For example, the text feature information of the prompt word is used as the key value of the attention module, the target reference feature information is used as the representative value of the attention module, and the output feature information of the kth network layer is used as the query value of the attention module. They are input into the attention module for fusion processing to obtain the input feature information of the k+1th network layer.

[0327] Based on the above steps, the electronic device can determine the feature information output by the last network layer of the denoising network. In one example, the electronic device can determine the feature information output by the last network layer of the denoising network as the i-th noise prediction value.

[0328] Next, the above step S204-B2 is performed to obtain the i+1th denoising feature information of the first image based on the i-th noise prediction value and the i-th denoising feature information of the first image. For example, the difference between the i-th denoising feature information of the first image and the i-th noise prediction value is determined as the i+1th denoising feature information of the first image.

[0329] The electronic device uses the i+1th denoising feature information of the first image determined above as the new i-th denoising feature information, returns to execute the above S204-B1 step, repeats it N times, obtains the N-th denoising feature information of the first image, and then uses the N-th denoising feature information as the denoised image feature information.

[0330] Finally, the electronic device executes the above step S204-C, that is, the electronic device decodes the denoised image feature information to obtain the first living body attack image.

[0331] In an embodiment of the present application, the above-mentioned denoised image feature information is feature information in the latent space, and the electronic device needs to decode the denoised image feature information to convert the denoised image feature information from the latent space to the pixel space to obtain the first living attack image.

[0332] In some embodiments, Fig.18 As shown, the electronic device decodes the denoised image feature information through a decoder (such as a VAE decoder) to obtain a first live attack image. The first live attack image is based on the prompt word and the target reference feature information, so that the generated first live attack image not only meets the requirements of the target live attack type indicated by the prompt word, but also includes the image detail features of the target live attack under the target live attack type, thereby improving the image generation effect.

[0333] The image processing method provided by the embodiment of the present application, by acquiring a first image and a prompt word, the prompt word is used to indicate the target live attack type, and the first image is a biometric image; then, the image feature extraction and texture feature extraction are performed on the first image to obtain the image feature information of the first image, and the text feature extraction is performed on the prompt word to obtain the text feature information of the prompt word, and the target reference feature information is determined, and the target reference feature information is used to represent the image feature of the target live attack, and the target live attack belongs to the target live attack type. Then, the text feature information of the prompt word and the target reference feature information are used as control conditions, and the image feature information of the first image is processed by the image generation model to obtain the first live attack image. It can be seen that when the embodiment of the present application generates the first live attack image based on the first image and the prompt word, the target reference feature information and the prompt word used to represent the image feature of the target live attack are used as control conditions to process the first image and generate the first live attack image, so that the generated first live attack image not only meets the requirements of the target live attack type indicated by the prompt word, but also includes the image features of the target live attack under the target live attack type, thereby improving the generation quality of the first live attack image.

[0334] The above is an overall introduction to the image processing method provided in the embodiment of the present application. The following is an introduction to the process of applying the image processing method in the embodiment of the present application to the generation of training samples.

[0335] Fig.19The flowchart of the training sample generation method provided in one embodiment of the present application is as follows. The execution subject of the embodiment of the present application can be a device having a function of generating training samples, such as a training sample generation device. The training sample generation device can be the above-mentioned Figure 1 The server 102 or terminal device 101 shown may also be Figure 1 The system is composed of the server 102 or the terminal device 101. For the convenience of description, the following embodiments take electronic devices as an example to illustrate the method of the embodiments of the present application.

[0336] like Fig.19 As shown, the training sample generation method of the embodiment of the present application includes:

[0337] S301: Acquire a fifth living body attack image and a prompt word.

[0338] This prompt word is used to indicate the target living attack type.

[0339] The training sample generation method in the embodiment of the present application takes the generation of training data for a living body detection model as an example.

[0340] In actual use, liveness detection is usually used in combination with other technologies, such as face authentication. Among them, liveness detection is the first line of defense and an important part of authentication security. Currently, face authentication has been applied in many businesses. For example, bank remote authentication, face payment, driver remote authentication, and access control systems.

[0341] In the process of remote account opening, in order to confirm the true identity of the account holder, face authentication technology is also used, and the liveness detection algorithm is also applied. The specific process is as follows: First, the object will obtain an image containing a face through the camera on the front end of the application. The front end transmits the image to the back end and calls the algorithm. The algorithm will perform liveness detection and return the result to the front end. If it is determined to be live, it passes, otherwise the authentication fails.

[0342] Facial authentication also plays an important role in the facial payment process. Liveness detection is an important part of controlling payment security. High-precision liveness detection methods can reject transactions attempted by some illegal attacks, ensure the security of transactions, and protect the interests of companies and individuals from damage.

[0343] In the access control system, in order to improve the efficiency of identity verification, the face image is directly obtained at the front end and sent to the packaged model for direct judgment and feedback of the results.

[0344] It can be seen that the accuracy of the liveness detection model is of great importance. The liveness detection model needs to be effectively trained so that it can achieve accurate detection. However, due to the complexity and variety of liveness attack types, relying solely on manpower to collect and label data to build training samples cannot meet actual business needs. The collected data may be homogenized and single, and it is impossible to build an effective liveness attack sample, resulting in inaccurate detection of the liveness detection model and the risk of identity infringement.

[0345] In order to solve this technical problem, the embodiment of the present application applies the above-mentioned image processing method to the generation of training samples, and proposes a training sample generation method, which processes the fifth live attack image and the prompt word through the trained image generation model to generate the sixth live attack image. Among them, the fifth live attack image and the sixth live attack image are both training samples of the liveness detection model, that is, through the method of the embodiment of the present application, the liveness attack samples of the liveness detection model can be expanded, and then the effective training of the liveness detection model can be realized.

[0346] The fifth live attack image may be a live attack image of a certain material, such as a paper copy, a screen copy, or a 3D head model and a 3D mask.

[0347] Exemplarily, the prompt word is used to indicate the target live body attack type. For example, if the fifth live body attack image is a 3D mask, the prompt word is “3D Mask”, “Silica gel”, indicating that the material of the 3D mask is changed to silica gel material.

[0348] The specific implementation process of the above S301 can refer to the relevant description of the above S201, which will not be repeated here.

[0349] S302: Perform image feature extraction and texture feature extraction on the fifth live body attack image to obtain image feature information of the fifth live body attack image and texture feature information of the fifth live body attack image.

[0350] In the embodiment of the present application, after the electronic device obtains the fifth liveness attack image and the prompt word corresponding to the fifth liveness attack image based on the above steps, in order to achieve mutual fusion between the image information of the fifth liveness attack image and the semantic information of the prompt word, the electronic device first extracts the image feature information of the fifth liveness attack image and the texture feature information of the fifth liveness attack image.

[0351] In some embodiments, before extracting image feature information of the fifth liveness attack image and before extracting texture feature information of the fifth liveness attack image, the electronic device first preprocesses the fifth liveness attack image to obtain a preprocessed fifth liveness attack image.

[0352] The specific manner of preprocessing the fifth live body attack image in the embodiment of the present application is not limited, for example, the clarity and sharpness of the fifth live body attack image are processed.

[0353] In one example, the electronic device crops the size of the fifth living body attack image to be consistent with the size of the input image of the image generation model based on the size of the input image of the image generation model.

[0354] In some examples, the electronic device performs a cutout process on the fifth liveness attack image to cut out the object region in the first image, and then adjusts the size of the object region to the first size. Then, the electronic device performs a center cropping of the object region of the first size to the second size to obtain a pre-processed fifth liveness attack image.

[0355] Next, the electronic device performs image feature extraction and texture feature extraction on the pre-processed fifth living body attack image.

[0356] like Fig. 20 As shown, the embodiment of the present application encodes the fifth live attack image through an image feature encoder to compress the fifth live attack image from the pixel space to the latent space to obtain image feature information of the fifth live attack image. The embodiment of the present application does not limit the specific network structure of the image feature encoding and decoding. Exemplarily, the image feature encoding and decoding can be the image feature encoding and decoding information in the VAE encoder or CLIP.

[0357] The embodiment of the present application does not limit the specific manner in which the electronic device extracts the texture feature information of the fifth living body attack image. For example, Fig. 20 As shown, the electronic device extracts texture feature information of the fifth live attack image through a texture feature extraction module. Optionally, the texture feature extraction module may include a gray level co-occurrence matrix, a Tamura texture feature, an autoregressive texture model, a wavelet transform, and the like.

[0358] The specific implementation process of the above S302 can refer to the relevant description of the above S202, which will not be repeated here.

[0359] S303: extract text features of the prompt word to obtain text feature information of the prompt word.

[0360] It should be noted that there is no particular order in which S302 and S303 are executed, that is, S302 may be executed before S303 or after S303, or may be executed synchronously with S303.

[0361] like Fig. 20As shown, the embodiment of the present application uses a text encoder to extract text features of the prompt word in the electronic device to obtain text feature information of the prompt word. The embodiment of the present application does not limit the specific network structure of the text encoder. For example, the text encoder can be a CLIP text encoder.

[0362] The specific implementation process of the above S303 can refer to the relevant description of the above S203, which will not be repeated here.

[0363] S304: Using the text feature information of the prompt word and the texture feature information of the fifth live body attack image as control conditions, the image feature information of the fifth live body attack image is processed by an image generation model to obtain a sixth live body attack image.

[0364] like Fig. 20 As shown, the image feature information of the fifth live attack image, the text feature information of the prompt word, and the texture feature information of the fifth live attack image are input into the image generation model, so that the image generation model processes the image feature information of the fifth live attack image with the text feature information of the prompt word and the texture feature information of the fifth live attack image as control conditions to generate the sixth live attack image. Since the texture feature information of the fifth live attack image and the text feature information of the prompt word are considered when generating the sixth live attack image, the generated sixth live attack image not only meets the requirements of the prompt word, but also includes the detailed features (such as texture features) of the fifth live attack image, thereby improving the generation effect of the sixth live attack image.

[0365] The embodiment of the present application does not limit the specific network structure of the above-mentioned image generation model.

[0366] In some embodiments, if the image generation model includes a denoising network, then the above S304 includes the following steps S304-A to S304-C:

[0367] S304-A, performing noise processing on the image feature information of the fifth live body attack image to obtain noise feature information of the fifth live body attack image;

[0368] S304-B, using the text feature information of the prompt word and the texture feature information of the fifth living body attack image as control conditions, denoising the noisy feature information of the fifth living body attack image through a denoising network to obtain denoised image feature information;

[0369] S304-C, decoding the denoised image feature information to obtain a sixth living body attack image.

[0370] In some embodiments, the above S304-B ​​includes the following steps S304-B1 to S304-B3:

[0371] S304-B1, using the text feature information of the prompt word and the texture feature information of the fifth living body attack image as control conditions, denoising the i-th denoised feature information of the fifth living body attack image through a denoising network to obtain an i-th noise prediction value, where i is a positive integer less than or equal to N, and if i is 1, the i-th denoised feature information is the noisy feature information of the fifth living body attack image;

[0372] S304-B2, obtaining (i+1) th denoising feature information of the fifth living attack image based on the i th noise prediction value and the i th denoising feature information of the fifth living attack image;

[0373] S304-B3, taking the (i+1)th denoising feature information as the new (i)th denoising feature information, returning to repeat the above steps, and obtaining the (N)th denoising feature information of the fifth living body attack image;

[0374] S304-B4, determining the Nth denoising feature information as the denoised image feature information.

[0375] The following introduces a specific process of using a denoising network to denoise the noisy feature information of the fifth living attack image with the text feature information of the prompt word and the texture feature information of the fifth living attack image as control conditions to obtain the denoised image feature information.

[0376] In the embodiment of the present application, the denoising process using the denoising network is an iterative process. For example, the denoising network is used to denoise the i-th denoised feature information of the fifth live attack image with at least one of the text feature information of the prompt word and the texture feature information of the fifth live attack image as a control condition to obtain the i-th noise prediction value. Then, based on the i-th noise prediction value and the i-th denoised feature information of the fifth live attack image, the i+1-th denoised feature information of the fifth live attack image is obtained. Then, the i+1-th denoised feature information is used as the new i-th denoised feature information, and the denoising network is used to denoise the new i-th denoised feature information with at least one of the text feature information of the prompt word and the texture feature information of the fifth live attack image as a control condition to obtain the i-th noise prediction value. Based on the i-th noise prediction value and the i-th denoised feature information of the fifth live attack image, the i+1-th denoised feature information of the fifth live attack image is obtained. Next, the i+1th denoising feature information is used as the new i-th denoising feature information and returned for execution. After repeating the denoising operation N times, the N-th denoising feature information can be obtained, and then the N-th denoising feature information is determined as the denoised image feature information.

[0377] The following is an introduction to the specific process of using the text feature information of the prompt word and the texture feature information of the fifth living attack image as control conditions in the above S304-B1, performing denoising on the i-th denoising feature information of the fifth living attack image through a denoising network, and obtaining the i-th noise prediction value.

[0378] In an embodiment of the present application, the i-th denoising feature information of the fifth live attack image is denoised by a denoising network, and at least one of the text feature information of the prompt word and the texture feature information of the fifth live attack image can be fused with the output feature information of at least part of the network layers in the denoising network, thereby achieving binding of the text feature information of the prompt word and the texture feature information of the fifth live attack image as conditions and the distribution of the fifth live attack image.

[0379] In the embodiment of the present application, the electronic device uses the text feature information of the prompt word and the texture feature information of the fifth living attack image as control conditions, performs denoising on the i-th denoising feature information of the fifth living attack image through a denoising network, and does not limit the specific method of obtaining the i-th noise prediction value.

[0380] In some embodiments, if the denoising network includes multiple network layers, the above S304-B1 includes the following steps S304-B11 and S304-B12:

[0381] S304-B11, using the i-th denoising feature information of the fifth living body attack image as input feature information of the denoising network;

[0382] S304-B12, the feature information output by some network layers of the denoising network is at least fused with the text feature information of the prompt word, and the feature information output by some network layers of the denoising network is at least fused with the texture feature information of the fifth living attack image to obtain the i-th noise prediction value.

[0383] In one example, the denoising network includes multiple network layers. The electronic device fuses the feature information output by some network layers of the denoising network with at least the text feature information of the prompt word, and fuses the feature information output by some network layers of the denoising network with at least the texture feature information of the fifth live attack image to obtain the i-th noise prediction value.

[0384] In one example, the electronic device fuses the feature information output by each network layer of the denoising network with the text feature information of the prompt word and the texture feature information of the fifth live attack image, and uses the fused feature information as the input feature information of the next network layer. By analogy, the feature information output by the last network layer of the denoising network is obtained as the i-th noise prediction value.

[0385] In one example, the electronic device fuses the output feature information of some network layers in the denoising network with the text feature information of the prompt word, and fuses the output feature information of some network layers of the denoising network with the texture feature information of the fifth live attack image. For example, the electronic device fuses the feature information 1 output by the first network layer of the denoising network with the text feature information of the prompt word, and uses the fused feature information as the input feature information of the second network layer. After fusing the output feature information 2 of the second network layer with the texture feature information of the fifth live attack image, it is used as the input feature information of the third network layer, and so on, to obtain the feature information output by the last network layer of the denoising network as the i-th noise prediction value.

[0386] In one example, the electronic device fuses the output feature information of some network layers in the denoising network with at least one of the text feature information of the prompt word and the texture feature information of the fifth live attack image, and fuses the output feature information of some network layers in the denoising network with the text feature information of the prompt word and the texture feature information of the fifth live attack image. For example, the electronic device fuses the feature information 1 output by the first network layer of the denoising network with the text feature information of the prompt word, and uses the fused feature information as the input feature information of the second network layer. After fusing the output feature information 2 of the second network layer with the texture feature information of the fifth live attack image, it is used as the input feature information of the third network layer. After fusing the output feature information 3 of the third network layer with the texture feature information of the fifth live attack image and the text feature information of the prompt word, it is used as the input feature information of the fourth network layer. By analogy, the feature information output by the last network layer of the denoising network is obtained as the i-th noise prediction value.

[0387] In the embodiment of the present application, in order to facilitate the description of the process of fusing the output feature information of the network layer in the denoising network with the text feature information of the prompt word and at least one of the texture feature information of the fifth live attack image, the kth network layer in the denoising network is used as an example for description. At this time, the above S304-B12 may include the following steps S304-B12-a and S304-B12-b:

[0388] S304-B12-a, for the kth network layer in the denoising network, fusing the output feature information of the kth network layer with at least one of the text feature information of the prompt word and the texture feature information of the fifth live attack image to obtain the input feature information of the k+1th network layer in the denoising network, where k is a positive integer;

[0389] S304-B12-b. Based on the input feature information of the k+1th network layer, obtain the i-th noise prediction value.

[0390] In an embodiment of the present application, for the kth network layer of the denoising network, the electronic device fuses the output feature information of the kth network layer with at least one of the text feature information of the prompt word and the texture feature information of the fifth live attack image to obtain the input feature information of the k+1th network layer in the denoising network. For example, the electronic device fuses the output feature information of the kth network layer with the text feature information of the prompt word, and uses the fused feature information as the input feature information of the k+1th network layer in the denoising network. For another example, the electronic device fuses the output feature information of the kth network layer with the texture feature information of the fifth live attack image, and uses the fused feature information as the input feature information of the k+1th network layer in the denoising network. For another example, the electronic device fuses the output feature information of the kth network layer with the texture feature information of the fifth live attack image and the text feature information of the prompt word, and uses the fused feature information as the input feature information of the k+1th network layer in the denoising network.

[0391] In some embodiments, the electronic device fuses the output feature information of the kth network layer with the text feature information of the prompt word and at least one of the texture feature information of the fifth live attack image through an attention mechanism to obtain the input feature information of the k+1th network layer.

[0392] Based on the above steps, the electronic device can determine the feature information output by the last network layer of the denoising network. In one example, the electronic device can determine the feature information output by the last network layer of the denoising network as the i-th noise prediction value.

[0393] Next, the above step S304 - B2 is performed to obtain the (i+1)th denoising feature information of the fifth living body attack image based on the i-th noise prediction value and the i-th denoising feature information of the fifth living body attack image.

[0394] The electronic device uses the i+1th denoising feature information of the fifth living body attack image determined above as the new i-th denoising feature information, returns to execute the above step S304-B1, repeats the execution N times, obtains the N-th denoising feature information of the fifth living body attack image, and then uses the N-th denoising feature information as the denoised image feature information.

[0395] Finally, the electronic device executes the above step S304-C, that is, the electronic device decodes the denoised image feature information to obtain a sixth living body attack image.

[0396] In some embodiments, Fig. 20As shown, the electronic device decodes the denoised image feature information through a decoder (such as a VAE decoder) to obtain a sixth live attack image. The sixth live attack image is based on the prompt word and the texture feature information of the fifth live attack image, and the initial features of the object in the fifth live attack image are transformed into the target features described by the prompt word, thereby improving the image generation effect.

[0397] The specific implementation process of the above S304 can refer to the relevant description of the above S204, which will not be repeated here.

[0398] The training sample generation method provided by the embodiment of the present application obtains a fifth live attack image and a prompt word, where the prompt word is used to indicate that the initial features of the object in the fifth live attack image are transformed into the target features described by the prompt word; then, image feature extraction and texture feature extraction are performed on the fifth live attack image to obtain image feature information of the fifth live attack image and texture feature information of the fifth live attack image, and text feature extraction is performed on the prompt word to obtain text feature information of the prompt word, and then the text feature information of the prompt word and the texture feature information of the fifth live attack image are used as control conditions, the image feature information of the fifth live attack image is processed by an image generation model to obtain a sixth live attack image, and the loss of the image generation model is determined based on the fifth live attack image and the sixth live attack image, and the image generation model is trained based on the loss. It can be seen that when the embodiment of the present application generates the sixth liveness attack image based on the fifth liveness attack image and the prompt word, the texture feature information and the prompt word of the fifth liveness attack image are used as control conditions to process the fifth liveness attack image to generate the sixth liveness attack image. The generated sixth liveness attack image not only includes the features described by the prompt word, but also retains the detailed features of the fifth liveness attack image, thereby improving the generation quality of the sixth liveness attack image.

[0399] Combination of the above Figures 2 to 20 , describes in detail the method embodiment of the present application, and the following is combined with Fig.21 , describe in detail the device embodiments of the present application.

[0400] Fig.21 It is a schematic block diagram of an image processing device provided in one embodiment of the present application.

[0401] like Fig.21 As shown, the image processing device 10 includes:

[0402] An acquisition unit 11 is used to acquire a first image and a prompt word, wherein the prompt word is used to indicate a target live body attack type, and the first image is a biometric feature image;

[0403] A feature extraction unit 12, used to extract image feature information of the first image and text feature information of the prompt word;

[0404] A determination unit 13, configured to determine target reference feature information, wherein the target reference feature information is used to represent image features of a target live body attack, and the target live body attack belongs to the target live body attack type;

[0405] The processing unit 14 is used to process the image feature information of the first image through an image generation model using the text feature information of the prompt word and the target reference feature information as control conditions to obtain a first live attack image for the biometric feature image.

[0406] In some embodiments, the determination unit 13 is specifically used to determine information of N candidate reference features corresponding to the target live attack type, and select information of a candidate reference feature from the information of the N candidate reference features as the target reference feature information, where the information of the N candidate reference features is image feature information of N types of live attacks belonging to the target live attack type, and N is a positive integer; or, obtain a second live attack image, and extract live attack feature information of the second live attack image as the target reference feature information.

[0407] In some embodiments, the information of the N candidate reference features or the live attack feature information of the second live attack image is image texture feature information.

[0408] In some embodiments, the image generation model includes a denoising network, and the processing unit 14 is specifically used to perform noise processing on the image feature information of the first image to obtain the noise feature information of the first image;

[0409] Taking the text feature information of the prompt word and the target reference feature information as control conditions, the denoising network is used to perform denoising processing on the i-th denoising feature information of the first image to obtain an i-th noise prediction value, where i is a positive integer less than or equal to N. If i is 1, the i-th denoising feature information is the noise feature information of the first image; based on the i-th noise prediction value and the i-th denoising feature information of the first image, the i+1-th denoising feature information of the first image is obtained; the i+1-th denoising feature information is used as the new i-th denoising feature information, and the above steps are repeated to obtain the N-th denoising feature information of the first image; the N-th denoising feature information is determined as the denoised image feature information; the denoised image feature information is decoded to obtain the first living attack image.

[0410] In some embodiments, the processing unit 14 is specifically used to use the i-th denoising feature information of the first image as the input feature information of the denoising network; fuse the feature information output by some network layers of the denoising network with at least the text feature information of the prompt word, and fuse the feature information output by some network layers of the denoising network with at least the target reference feature information to obtain the i-th noise prediction value.

[0411] In some embodiments, the processing unit 14 is specifically used to fuse the output feature information of the kth network layer in the denoising network with the text feature information of the prompt word and at least one of the target reference feature information to obtain the input feature information of the k+1th network layer in the denoising network, where k is a positive integer; and obtain the i-th noise prediction value based on the input feature information of the k+1th network layer.

[0412] In some embodiments, the processing unit 14 is specifically used to use the text feature information of the prompt word as the key value of the attention module, use the target reference feature information as the representative value of the attention module, and use the output feature information of the kth network layer as the query value of the attention module, input it into the attention module for fusion processing, and obtain the input feature information of the k+1th network layer.

[0413] In some embodiments, the feature extraction unit 12 is specifically configured to preprocess the first image to obtain a preprocessed first image; and extract image features from the preprocessed first image to obtain image feature information of the first image.

[0414] In some embodiments, the feature extraction unit 12 is specifically used to perform cutout processing on the first image to cut out the object area in the first image; adjust the size of the object area to a first size; center-crop the object area of ​​the first size to a second size, so as to obtain the preprocessed first image.

[0415] It should be understood that the device embodiment and the method embodiment may correspond to each other, and similar descriptions may refer to the method embodiment. To avoid repetition, they will not be described here. Specifically, Fig.21 The device shown can execute the above-mentioned image processing method embodiment, and the above-mentioned and other operations and / or functions of each module in the device are respectively for realizing the above-mentioned method embodiment, which will not be described in detail here for the sake of brevity.

[0416] Fig. 22 It is a schematic block diagram of a model training device provided in one embodiment of the present application.

[0417] like Fig. 22As shown, the model training device 20 includes:

[0418] An acquisition unit 21 is used to acquire a training image and a prompt word, wherein the prompt word is used to indicate a target live attack type, and the training image is a biometric image or a live attack image;

[0419] A feature extraction unit 22, used to extract image feature information of the training image and text feature information of the prompt word;

[0420] A determination unit 23, configured to determine target reference feature information, wherein the target reference feature information is used to represent image features of a target live body attack, and the target live body attack belongs to the target live body attack type;

[0421] The processing unit 24 is used to process the image feature information of the training image by an image generation model using the text feature information of the prompt word and the target reference feature information as control conditions to obtain a third live attack image for the biometric feature image;

[0422] The training unit 25 is used to determine the loss of the image generation model based on the third living body attack image and the target reference feature information, and train the image generation model based on the loss.

[0423] In some embodiments, the determination unit 23 is specifically used to determine information of N candidate reference features corresponding to the target live attack type, and select information of one candidate reference feature from the information of the N candidate reference features as the target reference feature information, where the information of the N candidate reference features is image feature information of N types of live attacks belonging to the target live attack type, and N is a positive integer; or, if the training image is a live attack image, extract the live attack feature information of the training image as the target reference feature information; or, obtain a fourth live attack image, extract the live attack feature information of the fourth live attack image as the target reference feature information.

[0424] In some embodiments, the training unit 25 is specifically configured to perform live attack feature extraction on the third live attack image to obtain live attack feature information of the third live attack image; and determine the loss of the image generation model based on the target reference feature information and the live attack feature information of the third live attack image.

[0425] In some embodiments, if the target reference feature information is the texture feature information of the training image, the training unit 25 is specifically used to extract texture features of the third live attack image to obtain texture feature information of the third live attack image; based on the texture feature information of the training image and the texture feature information of the third live attack image, determine the loss of the image generation model.

[0426] In some embodiments, the training unit 25 specifically determines a difference between the texture feature information of the training image and the texture feature information of the third living attack image; and determines a loss of the image generation model based on the difference.

[0427] In some embodiments, the feature extraction unit 22 is specifically used to preprocess the training image to obtain a preprocessed training image; and extract image features from the preprocessed training image to obtain image feature information of the training image.

[0428] In some embodiments, the feature extraction unit 22 is specifically used to perform cutout processing on the training image to cut out the object area in the training image; adjust the size of the object area to a first size; randomly crop the object area of ​​the first size to a second size to obtain the preprocessed training image.

[0429] It should be understood that the device embodiment and the method embodiment may correspond to each other, and similar descriptions may refer to the method embodiment. To avoid repetition, they will not be described here. Specifically, Fig. 22 The device shown can execute the above-mentioned model training method embodiment, and the aforementioned and other operations and / or functions of each module in the device are respectively for realizing the above-mentioned method embodiment, which will not be repeated here for the sake of brevity.

[0430] The above describes the device of the embodiment of the present application from the perspective of the functional module in conjunction with the accompanying drawings. It should be understood that the functional module can be implemented in hardware form, can be implemented by instructions in software form, and can also be implemented by a combination of hardware and software modules. Specifically, the steps of the method embodiment in the embodiment of the present application can be completed by the hardware integrated logic circuit and / or software form instructions in the processor, and the steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to perform, or a combination of hardware and software modules in the decoding processor to perform. Optionally, the software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory, and completes the steps in the above method embodiment in conjunction with its hardware.

[0431] Fig.23 It is a schematic block diagram of an electronic device provided in an embodiment of the present application, and the electronic device may be the above-mentioned terminal device or server.

[0432] like Fig.23 As shown, the electronic device 40 may include:

[0433] The memory 41 and the processor 42, the memory 41 is used to store the computer program 43 and transmit the computer program 43 to the processor 42. In other words, the processor 42 can call and run the computer program 43 from the memory 41 to implement the method in the embodiment of the present application.

[0434] For example, the processor 42 may be configured to execute the steps in the above method according to the instructions in the computer program 43 .

[0435] In some embodiments of the present application, the processor 42 may include but is not limited to:

[0436] General-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0437] In some embodiments of the present application, the memory 41 includes but is not limited to:

[0438] Volatile memory and / or non-volatile memory. Among them, the non-volatile memory can be read-only memory (ROM), programmable ROM (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM) or flash memory. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct memory bus random access memory (Direct Rambus RAM, DR RAM).

[0439] In some embodiments of the present application, the computer program 43 may be divided into one or more modules, which are stored in the memory 41 and executed by the processor 42 to complete the method for recording pages provided by the present application. The one or more modules may be a series of computer program instruction segments capable of completing specific functions, and the instruction segments are used to describe the execution process of the computer program 43 in the electronic device.

[0440] like Fig.23 As shown, the electronic device 40 may further include:

[0441] The transceiver 44 may be connected to the processor 42 or the memory 41 .

[0442] The processor 42 may control the transceiver 44 to communicate with other devices, specifically, to send information or data to other devices, or to receive information or data sent by other devices. The transceiver 44 may include a transmitter and a receiver. The transceiver 44 may further include an antenna, and the number of antennas may be one or more.

[0443] It should be understood that the various components in the electronic device 40 are connected via a bus system, wherein the bus system includes not only a data bus but also a power bus, a control bus and a status signal bus.

[0444] According to one aspect of the present application, a computer storage medium is provided, on which a computer program is stored, and when the computer program is executed by a computer, the computer can perform the method of the above method embodiment. In other words, the present application embodiment also provides a computer program product containing instructions, and when the instructions are executed by a computer, the computer can perform the method of the above method embodiment.

[0445] According to another aspect of the present application, a computer program product or computer program is provided, the computer program product or computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method of the above method embodiment.

[0446] In other words, when implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a digital video disc (DVD)), or a semiconductor medium (e.g., a solid state disk (SSD)), etc.

[0447] Those of ordinary skill in the art will appreciate that the modules and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0448] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the module is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0449] The modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. For example, each functional module in each embodiment of the present application may be integrated into a processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0450] The above contents are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An image processing method, characterized in that: include: Acquire a first image and a prompt word, wherein the prompt word is used to indicate a target live body attack type, and the first image is a biometric feature image; Extracting image feature information of the first image and text feature information of the prompt word; Determine information of N candidate reference features corresponding to the target live attack type, and select information of a candidate reference feature from the information of the N candidate reference features as target reference feature information, wherein the target reference feature information is used to represent image features of a target live attack, the target live attack belongs to the target live attack type, and the information of the N candidate reference features is image feature information of N types of live attacks belonging to the target live attack type, where N is a positive integer; The text feature information of the prompt word and the target reference feature information are used as control conditions, and the image feature information of the first image is processed by an image generation model to obtain a first live attack image for the biometric feature image.

2. The method according to claim 1, characterized in that The information of the N candidate reference features is image texture feature information.

3. The method according to claim 1, characterized in that The image generation model includes a denoising network, and the image feature information of the first image is processed by the image generation model using the text feature information of the prompt word and the target reference feature information as control conditions to obtain a first live attack image for the biometric feature image, including: Performing noise processing on the image feature information of the first image to obtain noise feature information of the first image; Using the text feature information of the prompt word and the target reference feature information as control conditions, the denoising network is used to perform denoising on the i-th denoised feature information of the first image to obtain an i-th noise prediction value, where i is a positive integer less than or equal to N, and if i is 1, the i-th denoised feature information is the noisy feature information of the first image; Based on the i-th noise prediction value and the i-th denoising feature information of the first image, obtaining (i+1)-th denoising feature information of the first image; The (i+1)th denoising feature information is used as new (i)th denoising feature information, and the process is repeated to obtain the (N)th denoising feature information of the first image. Determining the Nth denoising feature information as the denoised image feature information; The denoised image feature information is decoded to obtain the first live attack image for the biometric feature image.

4. The method according to claim 3, characterized in that The method uses the text feature information of the prompt word and the target reference feature information as control conditions, performs denoising processing on the i-th denoising feature information of the first image through the denoising network, and obtains the i-th noise prediction value, including: Using the i-th denoising feature information of the first image as input feature information of the denoising network; The feature information output by some network layers of the denoising network is at least fused with the text feature information of the prompt word, and the feature information output by some network layers of the denoising network is at least fused with the target reference feature information to obtain the i-th noise prediction value.

5. The method according to claim 4, characterized in that The step of fusing the feature information output by some network layers of the denoising network with at least the text feature information of the prompt word, and fusing the feature information output by some network layers of the denoising network with at least the target reference feature information, to obtain the i-th noise prediction value, includes: For the kth network layer in the denoising network, the output feature information of the kth network layer is fused with at least one of the text feature information of the prompt word and the target reference feature information to obtain the input feature information of the k+1th network layer in the denoising network, where k is a positive integer; Based on the input feature information of the k+1th network layer, the i-th noise prediction value is obtained.

6. The method according to claim 5, characterized in that The step of fusing the output feature information of the kth network layer with the text feature information of the prompt word and at least one of the target reference feature information to obtain the input feature information of the k+1th network layer in the denoising network includes: The text feature information of the prompt word is used as the key value of the attention module, the target reference feature information is used as the representative value of the attention module, and the output feature information of the kth network layer is used as the query value of the attention module. The information is input into the attention module for fusion processing to obtain the input feature information of the k+1th network layer.

7. A model training method, characterized in that: include: Acquire a training image and a prompt word, wherein the prompt word is used to indicate the target live attack type, and the training image is a biometric image or a live attack image; Extracting image feature information of the training image and text feature information of the prompt word; Determine information of N candidate reference features corresponding to the target live attack type, and select information of a candidate reference feature from the information of the N candidate reference features as target reference feature information, wherein the target reference feature information is used to represent image features of a target live attack, the target live attack belongs to the target live attack type, and the information of the N candidate reference features is image feature information of N types of live attacks belonging to the target live attack type, where N is a positive integer; Using the text feature information of the prompt word and the target reference feature information as control conditions, the image feature information of the training image is processed by an image generation model to obtain a third live attack image for the biometric feature image; Based on the third living body attack image and the target reference feature information, a loss of the image generation model is determined, and the image generation model is trained based on the loss.

8. The method according to claim 7, characterized in that The determining the loss of the image generation model based on the third living attack image and the target reference feature information includes: Extracting liveness attack features from the third liveness attack image to obtain liveness attack feature information of the third liveness attack image; Based on the target reference feature information and the live attack feature information of the third live attack image, a loss of the image generation model is determined.

9. The method according to claim 8, characterized in that If the target reference feature information is texture feature information of the training image, then extracting liveness attack features from the third liveness attack image to obtain liveness attack feature information of the third liveness attack image includes: Extracting texture features of the third live attack image to obtain texture feature information of the third live attack image; The determining the loss of the image generation model based on the target reference feature information and the live attack feature information of the third live attack image includes: Based on the texture feature information of the training image and the texture feature information of the third living body attack image, a loss of the image generation model is determined.

10. An image processing device, characterized in that: include: An acquisition unit, used for acquiring a first image and a prompt word, wherein the prompt word is used for indicating a target live body attack type, and the first image is a biometric feature image; A feature extraction unit, used to extract image feature information of the first image and text feature information of the prompt word; a determining unit, configured to determine information of N candidate reference features corresponding to the target living body attack type, and select information of a candidate reference feature from the information of the N candidate reference features as target reference feature information, wherein the target reference feature information is used to represent image features of a target living body attack, the target living body attack belongs to the target living body attack type, and the information of the N candidate reference features is image feature information of N types of living body attacks belonging to the target living body attack type, where N is a positive integer; The processing unit is used to process the image feature information of the first image through an image generation model using the text feature information of the prompt word and the target reference feature information as control conditions to obtain a first live attack image for the biometric feature image.

11. A model training device, characterized in that: include: An acquisition unit, used for acquiring a training image and a prompt word, wherein the prompt word is used for indicating a target live attack type, and the training image is a biometric image or a live attack image; A feature extraction unit, used to extract image feature information of the training image and text feature information of the prompt word; a determining unit, configured to determine information of N candidate reference features corresponding to the target living body attack type, and select information of a candidate reference feature from the information of the N candidate reference features as target reference feature information, wherein the target reference feature information is used to represent image features of a target living body attack, the target living body attack belongs to the target living body attack type, and the information of the N candidate reference features is image feature information of N types of living body attacks belonging to the target living body attack type, where N is a positive integer; a processing unit, configured to process the image feature information of the training image by using the text feature information of the prompt word and the target reference feature information as control conditions through an image generation model to obtain a third live attack image for the biometric feature image; A training unit is used to determine the loss of the image generation model based on the target reference feature information and the third living attack image, and train the image generation model based on the loss.

12. An electronic device, characterized in that: including a processor and a memory; The memory is used to store computer programs; The processor is used to execute the computer program to implement the method according to any one of claims 1 to 6 or 7 to 9.

13. A computer-readable storage medium, characterized in that: For storing computer programs; The computer program enables a computer to execute the method according to any one of claims 1 to 6 or 7 to 9 above.

Images