Facial feature processing method and device

Abstract

The embodiment of the invention provides a facial feature processing method and device. The method comprises the steps that first facial feature data of a first user are acquired, the first facial feature data correspond to a first facial image of the first user, and the face in the first facial image has a first expression; obtaining first expression feature data of a second user, wherein the first expression feature data corresponds to a second expression carried by the face in a second face image of the second user; generating a third facial image of the first user according to the first facial feature data and the first expression feature data by using an image generation model; and extracting second facial feature data from the third facial image, associating the second facial feature data with the first facial feature data, and adding the second facial feature data and the user identifier of the first user into a facial feature library.

Description

Technical Field

[0001] The embodiments of the present specification belong to the field of computer technology, and more particularly to a facial feature processing method and device. Background Art

[0002] Facial retrieval is widely used in scenarios including risky user identification, facial deduplication, etc. Currently, facial retrieval is usually performed using a facial feature library, which is generally constructed based on facial feature data of multiple collected facial images of users.

[0003] A reasonable and reliable solution is needed to optimize the facial feature library and achieve facial retrieval optimization. Summary of the invention

[0004] The purpose of the present invention is to provide a facial feature processing solution, which can optimize the facial feature library and realize facial retrieval optimization.

[0005] The first aspect of the present specification provides a facial feature processing method, including: obtaining first facial feature data of a first user, the first facial feature data corresponding to a first facial image of the first user, and the face in the first facial image has a first expression; obtaining first expression feature data of a second user, the first expression feature data corresponding to a second expression on the face in a second facial image of the second user; using an image generation model to generate a third facial image of the first user based on the first facial feature data and the first expression feature data; extracting second facial feature data of the third facial image, and associating the second facial feature data with the first facial feature data and a user identifier of the first user to add to a facial feature library.

[0006] According to a second aspect of the present specification, there is provided a facial feature processing device, comprising: a first acquisition unit, configured to acquire first facial feature data of a first user, wherein the first facial feature data corresponds to a first facial image of the first user, and the face in the first facial image has a first expression; a second acquisition unit, configured to acquire first expression feature data of a second user, wherein the first expression feature data corresponds to a second expression on the face in the second facial image of the second user; a generation unit, configured to generate a third facial image of the first user according to the first facial feature data and the first expression feature data using an image generation model; and a storage unit, configured to extract second facial feature data from the third facial image, and associate the second facial feature data with the first facial feature data and a user identifier of the first user and add the second facial feature data to a facial feature library.

[0007] A third aspect of the present specification provides a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the method described in the first aspect.

[0008] A fourth aspect of the present specification provides a computing device, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method described in the first aspect is implemented.

[0009] The fifth aspect of this specification provides a computer program product, comprising a computer program / instruction, which, when executed by a processor, implements the steps of the method described in the first aspect.

[0010] The solution provided by the above-mentioned embodiments of this specification can utilize an image generation model to generate a third facial image of the first user based on the first facial feature data of the first user and the first expression feature data of the second user. The first facial feature data corresponds to the first facial image of the first user, and the face in the first facial image has a first expression. The first expression feature data corresponds to the second expression on the face in the second facial image of the second user. It can be understood that the expression on the face in the third facial image is the same as or similar to the second expression. Afterwards, the solution can extract the second facial feature data of the third facial image, and by associating the second facial feature data with the first facial feature data and the user identification of the first user and adding it to the facial feature library, a facial feature data cluster of the first user is formed, thereby optimizing the facial feature library and realizing facial retrieval optimization. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the technical solutions of the embodiments of this specification, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0012] Figure 1 is a schematic diagram of the facial feature processing process in the embodiment of this specification;

[0013] Figure 2 is a flow chart of the facial feature processing method in the embodiment of this specification;

[0014] Figure 3 is another schematic diagram of the facial feature processing process in the embodiment of this specification;

[0015] Figure 4 is another flow chart of the facial feature processing method in the embodiment of this specification;

[0016] Figure 5 is an exemplary schematic diagram of the training process of the image generation model M in the embodiment of this specification;

[0017] Figure 6 It is a structural schematic diagram of a facial feature processing device in an embodiment of this specification. DETAILED DESCRIPTION

[0018] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this specification.

[0019] As mentioned above, face retrieval is widely used in scenarios including risky user identification, face deduplication, etc. Currently, face retrieval is usually performed using a facial feature library, which is generally constructed based on facial feature data of multiple collected facial images of users.

[0020] Considering that the facial feature library plays a key role in facial retrieval, the embodiments of this specification provide a facial feature processing solution, which can achieve facial retrieval optimization by optimizing the facial feature library, such as improving the recall rate of facial retrieval.

[0021] Figure 1 FIG. 1 is a schematic diagram of the facial feature processing process in the embodiment of this specification. Figure 1 As shown, the facial feature processing process involves facial feature data d1 of user A's facial image P1, expression feature data e1 of user B, image generation model M, and facial feature library D. Among them, the face in facial image P1 has a first expression, and the expression feature data e1 corresponds to the second expression of the face in user B's facial image P2, and the second expression is different from the first expression. The image generation model M can be a text-to-image generation model, and the text-to-image generation model includes but is not limited to a diffusion model, such as a Stable Diffusion model. The Stable Diffusion model is a text-to-image generation model based on deep learning.

[0022] In order to improve the ability of the facial feature library D to resist the expression of the user A that is the same as or similar to the second expression, the following can be done: Figure 1As shown in , the image generation model M is used to generate the facial image P3 of user A according to the facial feature data d1 and the expression feature data e1, and then the facial feature data d2 of the facial image P3 is extracted, and the facial feature data d2 is added to the facial feature library D by associating it with the facial feature data d1 and the user identification of user A to form a facial feature data cluster of user A, thereby optimizing the facial feature library D and realizing facial retrieval optimization.

[0023] According to the description above, face retrieval is widely used in scenarios including risky user identification and face deduplication. Taking the risky user identification scenario as an example, in this scenario, user A is specifically risky user A. In one example, risky user A can be a black industry user. Black industry users include but are not limited to users who use the Internet to conduct illegal and criminal activities.

[0024] As is known to all, the existing facial feature library is generally constructed based on the facial feature data of the collected facial images of multiple users. When the images of the multiple users are collected separately, the faces of the multiple users usually have similar expressions, which can be called normal expressions. Based on this, the first expression on the face in the facial image P1 of the risky user A can be called a normal expression. In addition, after the feature data of the face of the risky user with a normal expression is entered into the facial feature library for risky user identification, the risky user often tries to bypass the detection of the facial feature library by making more exaggerated expressions. In order to improve the ability of the facial feature library to resist the exaggerated expressions of risky users, it is necessary to obtain a facial image of the risky user with an exaggerated expression, and then add the facial feature data of the facial image to the facial feature library.

[0025] In the case where it is impossible to obtain a facial image of a risk user with an exaggerated expression by collecting images of the risk user, the solution provided in the embodiment of this specification proposes to generate a facial image of the risk user with an exaggerated expression by combining the feature data of the face with a normal expression of the risk user and the expression feature data of the exaggerated expressions of other users. Based on this, the second expression on the face in the facial image P2 of user B can be called an exaggerated expression. In the solution provided in the embodiment of this specification, the image generation model M can be used to generate a facial image P3 of the risk user A with an exaggerated expression according to the facial feature data d1 of the face with a normal expression of the risk user A and the expression feature data e1 of the exaggerated expression of user B. Afterwards, by extracting the facial feature data d2 of the facial image P3, and associating the facial feature data d2 with the facial feature data d1 and the user identification of the risk user A and adding it to the facial feature library D, a facial feature data cluster of the risk user A can be formed, and the ability of the facial feature library D to resist the exaggerated expression of the risk user A is improved, thereby realizing facial retrieval optimization.

[0026] Next, combine Figure 2 , introduces the facial feature processing process. Among them, Figure 2 This is a flow chart of the facial feature processing method in the embodiment of this specification. The method can be executed by any device, platform or device cluster with computing and processing capabilities, including steps S201-S207 as shown below.

[0027] like Figure 2 As shown, in step S201, the facial feature data d1 of user A is obtained, the facial feature data d1 corresponds to the facial image P1 of user A, and the face in the facial image P1 has a first expression.

[0028] Specifically, if the facial feature data d1 has been previously associated with the user ID of user A and added to the facial feature library D, the facial feature data d1 can be obtained from the facial feature library D. Alternatively, facial feature extraction can be performed on the facial image P1, for example Figure 3 As shown in , the facial feature extraction model is used to extract features from the facial image P1, thereby obtaining facial feature data d1. Figure 3 This is another schematic diagram of the facial feature processing process in the embodiment of this specification. The facial feature extraction model can be any existing machine learning model that can extract facial feature data in a facial image, and this embodiment of this specification does not specifically limit this.

[0029] In step S203 , the expression feature data e1 of user B is obtained, where the expression feature data e1 corresponds to the second expression on the face of user B in the facial image P2 .

[0030] Specifically, the pre-generated expression feature data e1 may be obtained. Alternatively, the expression feature data e1 may be obtained by extracting features from the facial image P2 using an expression extraction model. The expression extraction model includes but is not limited to: Figure 3 When the expression extraction model is a 3D expression extraction model, the expression feature data e1 is a 3D representation of the second expression of the face in the facial image P2.

[0031] In step S205, the facial image P3 of the user A is generated using the image generation model M according to the facial feature data d1 and the expression feature data e1.

[0032] Among them, the image generation model M can be a text-to-image generation model. In the case where the image generation model M is a text-to-image generation model, a text encoder can be used to generate a text encoding result based on the feature data d1, and then the image generation model M can be used to generate a facial image P3 based on the text encoding result and the expression feature data e1. The text encoder can be any existing encoder that can perform text encoding processing, for example, it can be BERT or CLIP (Contrastive Language-Image Pre-training), etc., which is not specifically limited here. BERT is a deep learning model based on Transformer, which aims to learn the deep representation of language through pre-training, so that it can adapt to a variety of natural language processing tasks. CLIP is a multimodal (text and image) pre-training model. Since BERT and CLIP are well-known technologies, they will not be described in detail here.

[0033] Furthermore, the input of the image generation model M may also include a target text for indicating the generated image (e.g. Figure 3 In this case, it can be as follows Figure 3 As shown in , a text encoder is used to generate a text encoding result according to the facial feature data d1 and the target text, and then an image generation model M is used to generate a facial image P3 according to the text encoding result and the expression feature data e1. The facial expression in the facial image P3 is the same as or similar to the second expression.

[0034] Next, in step S207, facial feature data d2 of the facial image P3 is extracted, and the facial feature data d2 is associated with the facial feature data d1 and the user identification of the user A and added to the facial feature library D.

[0035] Specifically, you can Figure 3 As shown in , the facial feature extraction model is used to extract features from the facial image P3 to obtain facial feature data d2. It should be noted that, when the facial feature data d1 has been previously associated with the user ID of user A and added to the facial feature library D, the facial feature data d2 is associated with the user ID and added to the facial feature library D; when the facial feature data d1 has not been previously added to the facial feature library D, both the facial feature data d2 and the facial feature data d1 are associated with the user ID and added to the facial feature library D. As a result, the facial feature data d2 and the facial feature data d1 can form a facial feature data cluster of user A in the facial feature library D.

[0036] Figure 2The solution provided by the corresponding embodiment can combine the facial feature data d1 of the face of user A with a first expression and the expression feature data e1 of the second expression of the second user to generate a facial image P3 of user A, and then form a facial feature data cluster of user A by associating the facial feature data d2 of the facial image P3 with the facial feature data d1 and the user identification of user A, thereby optimizing the facial feature library D and achieving facial retrieval optimization.

[0037] In one embodiment, the image generation model M may include a network M1 for generating an image, and a network M2 for controlling the image generation process. In one example, the network M1 may include Unet, and the network M2 may include ControlNet. Among them, Unet is a neural network with an Encoder-Decoder structure, which can be used to process the deep features of the image, help the model understand the image content and generate high-quality images. ControlNet is an auxiliary network for optimizing and improving the image generation model M, which is used to improve the performance of the image generation model M and provide more refined control capabilities.

[0038] Next, combine Figure 4 , introduces the facial feature processing process when the image generation model M includes a network M1 and a network M2. Among them, Figure 4 is another flow chart of the facial feature processing method in the embodiment of this specification. The method can be executed by any device, platform or device cluster with computing and processing capabilities, including steps S401-S409 as shown below.

[0039] like Figure 4 As shown, in step S401, the facial feature data d1 of user A is obtained, the facial feature data d1 corresponds to the facial image P1 of user A, and the face in the facial image P1 has a first expression.

[0040] In step S403 , the expression feature data e1 of user B is obtained, and the expression feature data e1 corresponds to the second expression on the face of user B in the facial image P2 .

[0041] Among them, for the explanation of steps S401 and S403, please refer to the relevant explanation of steps S201 and S203 in the previous text, which will not be repeated here.

[0042] Next, in step S405, the expression feature data e1 is processed using the network M2 in the image generation model M to obtain a processing result R1; wherein the image generation model M also includes the network M1 for generating images, and the network M2 is used to control the image generation process.

[0043] Next, in step S407, the network M1 is used to generate a facial image P3 of the user A according to the processing result R1 and the feature data d1.

[0044] As an example, the expression feature data e1 can be input into the second network for processing to obtain the processing result R1. After that, the processing result R1 and the facial feature data D1 can be input into the first network, so that the first network fuses the processing result R1 and the facial feature data D1, and generates a facial image P3 of user A according to the fusion result. The facial expression in the facial image P3 is the same as or similar to the second expression.

[0045] Next, in step S409, facial feature data d2 of the facial image P3 is extracted, and the facial feature data d2 is associated with the facial feature data d1 and the user identification of the user A and added to the facial feature library D.

[0046] Among them, for the explanation of step S409, please refer to the relevant description of step S207 in the previous text, which will not be repeated here.

[0047] Figure 4 The scheme provided by the corresponding embodiment adopts an image generation model M including a network M1 for generating images and a network M2 for controlling the image generation process. In this scheme, the network M2 can be used to process the expression feature data e1 of the second expression of user B to obtain the processing result R1, and then the network M1 is used to generate the facial image P3 of user A according to the processing result R1 and the facial feature data d1 of the face of user A with the first expression, so that the network M2 can control the generation process of the facial image P3, ensure that the face in the facial image P3 is the face of user A, and ensure the image generation quality, avoiding false interruptions during the retrieval process. In addition, the facial feature data d2 of the facial image P3 can be associated with the facial feature data d1 and the user identification of user A and added to the facial feature library D to form a facial feature data cluster of user A, thereby optimizing the facial feature library D and realizing facial retrieval optimization.

[0048] In one embodiment, the image generation model M can be implemented based on a diffusion model, and the first network as described above can be specifically used to predict noise. Figure 5 , introduces the training process of the image generation model M when the first network is specifically used to predict noise. Among them, Figure 5 It is an exemplary schematic diagram of the training process of the image generation model M in the embodiment of this specification.

[0049] like Figure 5As shown, the facial feature data d3 of the facial image P4 of the sample user C can be obtained, and the expression feature data e2 of the second expression on the face in the facial image x0 of the sample user C can be obtained; wherein the face in the facial image P4 has the first expression. It should be noted that the facial feature data d3 and the expression feature data e2 can be generated in advance; or, the facial feature extraction model as described above can be used to extract features from the facial image P4 to obtain the facial feature data d3, and the expression extraction model as described above can be used to extract features from the facial image x0 to obtain the expression feature data e2.

[0050] In addition, the noise image x of the facial image x0 at time step t can be generated based on the facial image x0, the sampled time step t and the noise ∈ that obeys the standard normal distribution. t . Wherein, t is a natural number in [1, T], and T is a natural number greater than or equal to 1. In an example, the facial image P4 and the facial image x0 are in the same training data, and the time step t and the noise ∈ can be sampled for the training data.

[0051] In practice, the following formula (1) can be used to calculate x t :

[0052]

[0053] Among them, ∈~N(0,1), N(0,1) represents a normal distribution with 0 as mean and 1 as standard deviation, that is, a standard normal distribution. t =1-β t , β t It is possible to express a schedule that controls the degree of regularization of the model.

[0054] Next, the second network may be used to process the expression feature data e2 to obtain a processing result R2.

[0055] Next, we can use the first network to generate the noise image x t , time step t, processing result R2 and facial feature data d3, predict noise ∈′. Where noise ∈′ is the noise image x predicted by the first network t Noise added in.

[0056] Next, the training loss can be determined based on the difference between the real noise ∈ and the predicted noise ∈′, and the parameters of the image generation model M can be adjusted with the goal of minimizing the training loss.

[0057] Specifically, the training loss can be calculated using the following formula (2):

[0058]

[0059] Among them, L can represent the loss function, ∈ θ The image generation model M can be represented by ∈ θ The output of is noise ∈.

[0060] When the image generation model M adopts the training process as described above, Figure 4 In a corresponding embodiment, the implementation process of step S407 may include: sampling a time step T and a noise image x at the time step T T , where T is a natural number greater than 0; the natural numbers from T to 1 are taken as the time step t in sequence, and the first network is used to generate the noise image x at the time step t and the time step t. t , processing result R1 and facial feature data d1, prediction noise ∈ t-1 , and according to the noise ∈ t-1 For the noisy image x t Perform denoising to obtain image x t-1 Based on the obtained image x0, the facial image P3 is determined. The image x0 can be determined as the facial image P3.

[0061] x t-1 It can be calculated by the following formula (3):

[0062]

[0063] It should be noted that in the training process and actual application of the image generation model M, the input of the first network may include an embedded representation of the time step t.

[0064] In one embodiment, in order to improve the model training efficiency, the image generation model M can be implemented based on the stable diffusion model. When the image generation model M is implemented based on the stable diffusion model, in the training process as described above, the facial image x0 of the sample user C can be encoded using an image encoder to obtain low-dimensional compressed data z0, and then based on z0, the sampled time step t and the noise ∈ that obeys the standard normal distribution, the noise image x0 of the facial image x0 at time step t is generated. t The compressed data z t . Subsequently, the first network can be used to compress the data z t , time step t, processing result R2 and facial feature data d3, predict noise ∈′, determine the training loss based on the difference between the real noise ∈ and the predicted noise ∈′, and adjust the parameters of the image generation model M with the goal of minimizing the training loss. It should be noted that replacing x0 with z0 in the above formulas (1) and (2), and replacing x t Replace with zt , we can get z t The calculation formula for , and the loss function used to calculate the training loss.

[0065] Accordingly, in the actual application stage of the image generation model M, such as in the implementation process of the above step S407, the image encoder can be used to generate the noise image x T Perform encoding processing to obtain low-dimensional compressed data z T After that, the natural numbers from T to 1 can be used as the time step t in sequence, and the first network can be used based on the time step t and the compressed data z t , processing result R1 and facial feature data d1, prediction noise ∈ t-1 , and according to the noise ∈ t-1 For compressed data z t Perform denoising to obtain compressed data z t-1 It should be noted that, as in the above formula (3), t-1 Replace with z t-1 , and x t Replace with z t , we can get z t-1 The calculation formula can be used to calculate z t-1 After the compressed data z0 is obtained, the compressed data z0 may be decoded by using an image decoder to obtain an image x0, and the facial image P3 may be determined based on the obtained image x0.

[0066] In one embodiment, after the facial feature data d2 is associated with the facial feature data d1 and the user identification of user A and added to the facial feature library D as described above, the facial feature data d1 and the facial feature data d2 of user A can be used for the user identification process. The execution subject of the user identification process can be any device, platform or device cluster with computing and processing capabilities. The execution subject can be the same as or different from the execution subject of the facial feature processing process described above, and no specific limitation is made here.

[0067] As an example, in the user identification process, a target facial image of the target user to be identified can be obtained, and then the target facial feature data of the target facial image can be extracted, and the target facial feature data can be matched with the facial feature data in the facial feature library D. Specifically, when matching the target facial feature data with the facial feature data d1 and the facial feature data d2 of user A in the facial feature library D, the target facial feature data can be first matched with one of the facial feature data d1 and the facial feature data d2, for example, the similarity between the target facial feature data and the one facial feature data is calculated, and if the similarity exceeds a preset threshold, it is determined that the target facial feature data and the one facial feature data match; otherwise, it is determined that the target facial feature data and the one facial feature data do not match. When it is determined that the target facial feature data and the one facial feature data match, it can be determined that the target user is user A.

[0068] When it is determined that the target facial feature data and the one facial feature data do not match, the target facial feature data can then be matched with the other facial feature data in the facial feature data d1 and the facial feature data d2. If the target facial feature data and the other facial feature data match, the target user can be determined to be user A. In addition, in the case that the target facial feature data and the other facial feature data do not match, if there is facial feature data in the facial feature library D that has not yet participated in the matching operation, the matching operation will continue to be performed based on the facial feature data and the target facial feature data; if there is no facial feature data in the facial feature library D that has not yet participated in the matching operation, a detection result indicating that the target user has not been detected can be generated. In summary, if the target facial feature data matches one of the facial feature data d1 and the facial feature data d2, the target user is determined to be user A.

[0069] Figure 6 FIG. 1 is a schematic diagram of the structure of the facial feature processing device in the embodiment of this specification. The device can be applied to any device, platform or device cluster with computing and processing capabilities. The device can perform the following steps: Figure 2 , Figure 4The described method includes: a first acquisition unit 601, configured to acquire first facial feature data of a first user, the first facial feature data corresponding to a first facial image of the first user, and the face in the first facial image has a first expression; a second acquisition unit 602, configured to acquire first expression feature data of a second user, the first expression feature data corresponding to a second expression on the face in a second facial image of the second user; a generation unit 603, configured to generate a third facial image of the first user based on the first facial feature data and the first expression feature data using an image generation model; a storage unit 604, configured to extract second facial feature data from the third facial image, and associate the second facial feature data with the first facial feature data and the user identification of the first user and add the second facial feature data to a facial feature library.

[0070] The present specification also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed in a computer, the computer is caused to execute the following Figure 2 , Figure 4 Describe the method.

[0071] The embodiment of the present specification also provides a computing device, including a memory and a processor, wherein the memory stores an executable code, and when the processor executes the executable code, the following is implemented: Figure 2 , Figure 4 Describe the method.

[0072] The embodiments of the present specification also provide a computer program product, including a computer program / instruction, which is executed by a processor to implement the following Figure 2 , Figure 4 Describe the steps of the method.

[0073] In the 1990s, improvements to a technology could be clearly distinguished as hardware improvements (for example, improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the method flow). However, with the development of technology, many improvements to the method flow today can be regarded as direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that an improvement in a method flow cannot be implemented using a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to ask a chip manufacturer to design and produce a dedicated integrated circuit chip. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0074] The controller can be implemented in any appropriate manner, for example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (such as software or firmware) that can be executed by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in a purely computer-readable program code manner, the controller can be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, this controller can be considered as a hardware component, and the devices included therein for implementing various functions can also be regarded as structures within the hardware component. Or even, the devices for implementing various functions can be regarded as both software modules for implementing the method and structures within the hardware component.

[0075] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a server system. Of course, the present application does not exclude that with the development of computer technology in the future, the computer that implements the functions of the above embodiments may be, for example, a personal computer, a laptop computer, a vehicle-mounted human-computer interaction device, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0076] Although one or more embodiments of the present specification provide method operation steps as described in the embodiments or flow charts, more or less operation steps may be included based on conventional or non-creative means. The order of steps listed in the embodiments is only one way of executing the order of many steps, and does not represent the only execution order. When the device or terminal product in practice is executed, it can be executed in sequence or in parallel according to the method shown in the embodiments or the drawings (for example, a parallel processor or a multi-threaded processing environment, or even a distributed data processing environment). The term "include", "include" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, product or equipment including a series of elements includes not only those elements, but also includes other elements that are not explicitly listed, or also includes elements inherent to such a process, method, product or equipment. In the absence of more restrictions, it is not excluded that there are other identical or equivalent elements in the process, method, product or equipment including the elements. For example, if the words first, second, etc. are used to represent the name, they do not represent any specific order.

[0077] For the convenience of description, the above devices are described in various modules according to their functions. Of course, when implementing one or more of the present specification, the functions of each module can be implemented in the same or more software and / or hardware, or the module implementing the same function can be implemented by a combination of multiple sub-modules or sub-units, etc. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0078] The present invention is described with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0079] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0080] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0081] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0082] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0083] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage, graphene storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0084] It should be understood by those skilled in the art that one or more embodiments of the present specification may be provided as a method, system or computer program product. Therefore, one or more embodiments of the present specification may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, one or more embodiments of the present specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0085] One or more embodiments of the present specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. One or more embodiments of the present specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0086] Each embodiment in this specification is described in a progressive manner, and the same and similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. In the description of this specification, the description of the reference terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of this specification. In this specification, the schematic representation of the above terms does not necessarily target the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples without contradiction.

[0087] The above description is only an example of one or more embodiments of the present specification and is not intended to limit one or more embodiments of the present specification. For those skilled in the art, one or more embodiments of the present specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present specification shall be included in the scope of the claims.

Claims

1. A facial feature processing method, comprising: Acquire first facial feature data of a first user, where the first facial feature data corresponds to a first facial image of the first user, where the face in the first facial image has a first expression; Acquire first expression feature data of a second user, where the first expression feature data corresponds to a second expression on the face of the second user in a second facial image; Generate a third facial image of the first user according to the first facial feature data and the first expression feature data using an image generation model; Second facial feature data of the third facial image is extracted, and the second facial feature data is associated with the first facial feature data and the user identification of the first user and added to a facial feature library.

2. The method according to claim 1, further comprising: Using a text encoder, generating a text encoding result according to the first facial feature data; The step of generating a third facial image of the first user according to the first facial feature data and the first expression feature data includes: The third facial image is generated according to the text encoding result and the first expression feature data.

3. The method according to claim 2, wherein: The step of generating a text encoding result according to the first facial feature data comprises: A text encoding result is generated according to the first facial feature data and a target text, wherein the target text is used to indicate the generation of an image.

4. The method according to claim 1, wherein: The obtaining of first facial feature data of the first user includes: Extracting features from the first facial image using a facial feature extraction model to obtain the first facial feature data; The extracting the second facial feature data of the third facial image comprises: The facial feature extraction model is used to perform feature extraction on the third facial image to obtain the second facial feature data.

5. The method according to claim 1, wherein: The obtaining of the first expression feature data of the second user includes: The expression extraction model is used to perform feature extraction on the second facial image to obtain the first expression feature data.

6. The method according to claim 5, wherein: The expression extraction model is a 3D expression extraction model, and the first expression feature data is a 3D representation of a second expression on the face in the second facial image.

7. The method according to any one of claims 1 to 6, wherein: The image generation model includes a first network for generating an image, and a second network for controlling the image generation process; The step of using the image generation model to generate a third facial image of the first user according to the first facial feature data and the first expression feature data includes: Processing the first expression feature data using the second network to obtain a first processing result; The third facial image is generated by using the first network according to the first processing result and the first facial feature data.

8. The method according to claim 1, wherein: The image generation model includes a first network for predicting noise, and a second network for controlling the image generation process; The training process of the image generation model includes: Acquire third facial feature data of a fourth facial image of a sample user, and acquire second expression feature data of a second expression on the face of a fifth facial image of the sample user; wherein the face in the fourth facial image has a first expression; Generate a noise image of the fifth facial image at time step t according to the fifth facial image, the sampled time step t and the first noise obeying the standard normal distribution; Processing the second expression feature data using the second network to obtain a second processing result; Using the first network, predicting a second noise based on the noise image, the time step t, the second processing result and the third feature data; determining a training loss based on a difference between the first noise and the second noise; The parameters of the image generation model are adjusted with the goal of minimizing the training loss.

9. The method according to claim 1, wherein: After associating the second facial feature data with the first facial feature data and the user identification of the first user and adding them into a facial feature library, the method further includes: Acquire a target facial image of a target user to be identified; Extracting target facial feature data of the target facial image; If the target facial feature data matches one of the first facial feature data and the second facial feature data, the target user is determined to be the first user.

10. A facial feature processing device, comprising: A first acquisition unit is configured to acquire first facial feature data of a first user, where the first facial feature data corresponds to a first facial image of the first user, where the face in the first facial image has a first expression; A second acquisition unit is configured to acquire first expression feature data of a second user, where the first expression feature data corresponds to a second expression on the face of the second user in a second facial image; a generating unit configured to generate a third facial image of the first user according to the first facial feature data and the first expression feature data by using an image generation model; The storage unit is configured to extract second facial feature data from the third facial image, and associate the second facial feature data with the first facial feature data and the user identification of the first user to add the second facial feature data into a facial feature library.

11. A computer program product, comprising a computer program / instruction, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 9.

12. A computing device comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method according to any one of claims 1 to 9 is implemented.

Images