Training method and generation method of senescence image generation model and electronic equipment

Abstract

The embodiment of the invention relates to the technical field of computer vision, and discloses a training method and a generation method of an aging image generation model, and electronic equipment, and the method comprises the steps: obtaining face images of different age values and text information corresponding to the face images, inputting the text information to a multi-modal pre-training network, obtaining text coding features, and carrying out the recognition of the text coding features; and inputting the face image and the text coding features into the senescence generation network to obtain a face image feature map, and inputting the face image feature map into the multi-modal pre-training network to obtain the senescence image, so that the generation capability of the senescence image generation model can be improved, the model can learn more senescence change features, and the user experience is improved. Therefore, the difference of senescence changes of different faces is adapted, and the senescence image generation effect is enhanced.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer vision technology, and in particular to a training method, a generation method, and an electronic device for an aging image generation model. Background Art

[0002] Facial aging image prediction has a wide range of applications in various fields. For example, in the field of criminal investigation, facial aging images can help the police better understand the changes in the suspect's appearance and provide clues for solving the case. Although medicine believes that an individual's facial aging changes with age, the results depend on the individual's psychological state and the physiological functions of aging. In the field of computer vision, the goal of computer-assisted facial aging is to provide facial images of a specific age with natural aging characteristics.

[0003] At present, large models are mostly used to generate aging images of different ages, which are used as training data. An end-to-end network structure is then used to train the model. Since large models tend to extract universal features rather than individual differences when dealing with complex aging changes, the generated aging images lack specificity, which in turn leads to little difference in the aging features learned by the model. Different faces generate the same aging change features, resulting in a lack of specificity in the generated aging images. Summary of the invention

[0004] The embodiments of the present application provide a training method, a generation method, and an electronic device for an aging image generation model to solve the problem that aging images generated by current network models lack specificity. The model can learn more aging change features to adapt to the differences in aging changes of different faces and enhance the image generation capability of the aging image generation model.

[0005] In a first aspect, an embodiment of the present application provides a training method for an aging image generation model, wherein the aging image generation model includes: a multimodal pre-training network and an aging generation network, and the method includes:

[0006] Obtaining a training data set, wherein the training data set includes face images of different age values ​​and text information corresponding to the face images;

[0007] Input the text information into the multimodal pre-trained network to obtain the text encoding features corresponding to the text information;

[0008] Input the face image and text encoding features into the aging generation network to obtain a face image feature map;

[0009] Input the facial image feature map into the multimodal pre-trained network to obtain an aging image;

[0010] A model loss function is constructed, and the aging image generation model is trained based on the training data set until the model loss function converges to generate a trained aging image generation model, wherein the model loss function includes a similarity loss.

[0011] In some embodiments, the multimodal pre-trained network includes a multimodal text encoding network;

[0012] Input the text information into the multimodal pre-trained network to obtain the text encoding features corresponding to the text information, including:

[0013] The text information is input into a multimodal text encoding network, and feature encoding is performed through the multimodal text encoding network to generate text encoding features corresponding to the text information, wherein the text information includes a real age value and a predicted age value of a face image.

[0014] In some embodiments, the aging generation network includes a downsampling module, an intermediate module, and an upsampling module;

[0015] The face image and text encoding features are input into the aging generation network to obtain the face image feature map, including:

[0016] Inputting the face image into a downsampling module for feature extraction to obtain a first image feature map;

[0017] Inputting the first image feature map and the text encoding features into the intermediate module for feature fusion to obtain a second image feature map;

[0018] The second image feature map and the text encoding features are input into the upsampling module to obtain a face image feature map.

[0019] In some embodiments, the intermediate module includes a first intermediate layer and a second intermediate layer, the first intermediate layer includes at least two sequentially connected feature extraction layers, and the second intermediate layer includes at least two sequentially connected feature fusion layers;

[0020] The first image feature map and the text encoding feature are input into the intermediate module for feature fusion to obtain the second image feature map, including:

[0021] Inputting the first image feature map into the first intermediate layer for feature extraction to obtain a high-level semantic feature map;

[0022] The high-level semantic feature map and text encoding features are input into the second intermediate layer for feature fusion to obtain the second image feature map.

[0023] In some embodiments, the downsampling module includes at least four sequentially connected downsampling layers, each of which is used to perform a downsampling operation on the face image to obtain at least four first image feature maps of different scales;

[0024] The upsampling module includes at least four upsampling layers connected in sequence, and each upsampling layer is used to perform upsampling operations on the second image feature map and the text encoding features to obtain at least four facial image feature maps of different scales.

[0025] In some embodiments, the multimodal pre-trained network includes: a multimodal image encoding network;

[0026] Input the facial image feature map into the multimodal pre-trained network to obtain an aging image, including:

[0027] The face image feature map is input into the multimodal image coding network, and feature encoding is performed through the multimodal image coding network to generate an aging image.

[0028] In some embodiments, the high-level semantic feature map and the text encoding feature are input into the second intermediate layer for feature fusion to obtain a second image feature map, including:

[0029]

[0030] Among them, F is the text encoding feature, x is the high-level semantic feature map, σ(F) is the mean of the text encoding feature, σ(x) is the mean of the high-level semantic feature map, μ(x) is the variance value of the high-level semantic feature map, and μ(F) is the variance value of the text encoding feature.

[0031] In some embodiments, the text encoding features include forward text features and reverse text features;

[0032] Similarity loss, including:

[0033]

[0034] Among them, Loss is the similarity loss, L 正 is the similarity loss between the positive text feature and the face image, L 反 is the similarity loss between the inverse text features and the face image.

[0035] In some embodiments, the similarity loss between the positive text and the face image includes:

[0036]

[0037] Among them, L 正 is the similarity loss between the positive text and the face image, F 1 is the positive text feature, F I is a face image;

[0038] The similarity loss between reverse text and face images includes:

[0039]

[0040] Among them, L 反 is the similarity loss between the positive text and the face image, F′ 1 is the reverse text feature, F I A face image.

[0041] In a second aspect, an embodiment of the present application provides a method for generating an aging image, the method comprising:

[0042] Acquire a face image and text information, wherein the text information includes a real age value and a predicted age value of the face image;

[0043] The face image and text information are input into an aging image generation model to obtain an aging image corresponding to the face image, wherein the aging image generation model is trained based on the method of the first aspect.

[0044] In a third aspect, an embodiment of the present application provides an electronic device, including:

[0045] at least one processor, and

[0046] a memory communicatively coupled to at least one processor, wherein:

[0047] The memory stores instructions that can be executed by at least one processor, and the instructions are executed by at least one processor so that the at least one processor can execute the method provided in the first aspect or the second aspect as described above.

[0048] Beneficial effects of the embodiments of the present application: Different from the prior art, the embodiments of the present application provide a training method for an aging image generation model, wherein the aging image generation model includes: a multimodal pre-trained network, an aging generation network, and the method includes: obtaining a training data set, wherein the training data set includes face images of different age values ​​and text information corresponding to the face images; inputting the text information into the multimodal pre-trained network to obtain text encoding features corresponding to the text information; inputting the face image and the text encoding features into the aging generation network to obtain a face image feature map; inputting the face image feature map into the multimodal pre-trained network to obtain an aging image; constructing a model loss function, and training the aging image generation model based on the training data set until the model loss function converges to generate a trained aging image generation model, wherein the model loss function includes a similarity loss.

[0049] On the one hand, by obtaining face images of different age values ​​and text information corresponding to the face images, inputting the text information into a multimodal pre-trained network to obtain text encoding features, inputting the face images and text encoding features into an aging generation network to obtain a face image feature map, and inputting the face image feature map into a multimodal pre-trained network to obtain an aging image, the present application can improve the generation capability of an aging image generation model, so that the model can learn more aging change features to adapt to the differences in aging changes of different faces, and enhance the image generation capability of the aging image generation model.

[0050] On the other hand, by constructing a model loss function for model training, the model loss function includes a similarity loss to obtain a trained aging image generation model. By training with similarity loss, the present application can enhance the generation effect of aging images. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] One or more embodiments are exemplarily described by pictures in the corresponding drawings, and these exemplified descriptions do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings represent similar elements, and unless otherwise stated, the figures in the drawings do not constitute proportional limitations.

[0052] Figure 1 It is a schematic diagram of an application environment of a training method for an aging image generation model provided in an embodiment of the present application;

[0053] Figure 2 is a structural schematic diagram of an aging image generation model provided in an embodiment of the present application;

[0054] Figure 3 It is a flowchart of a training method for an aging image generation model provided in an embodiment of the present application;

[0055] Figure 4 is a schematic diagram of key points of a face provided in an embodiment of the present application;

[0056] Figure 5 It is a schematic diagram of the structure of an aging generation network provided in an embodiment of the present application;

[0057] Figure 6 yes Figure 3 A detailed flowchart of step S303 in FIG.

[0058] Figure 7 is a structural diagram of a downsampling module provided in an embodiment of the present application;

[0059] Figure 8 It is a structural schematic diagram of an intermediate module provided in an embodiment of the present application;

[0060] Fig. 9 yes Figure 6 A detailed flowchart of step S3302 in FIG.

[0061] Fig.10 is a structural diagram of an upsampling module provided in an embodiment of the present application;

[0062] Fig.11 is a schematic diagram of generating an aging image provided by an embodiment of the present application;

[0063] Fig.12 It is a schematic diagram of an overall process of generating an aging image provided by an embodiment of the present application;

[0064] Fig.13 It is a flowchart of a method for generating an aging image provided in an embodiment of the present application;

[0065] Fig.14 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application.

[0066] Description of Figure Numbers:

[0067] Label name Label name 100 Application Environment 2214 Fourth downsampling layer 101 terminal 2022 Intermediate Module 102 server 2221 First middle layer 200 Generative Model for Aging Images 2222 Second middle layer 201 Multimodal pre-trained networks 2023 Upsampling module 211 Multimodal Text Encoding Network 2231 First upsampling layer 212 Multimodal Image Coding Network 2232 Second upsampling layer 202 Senescence generation network 2233 The third upsampling layer 2021 Downsampling module 2234 Fourth upsampling layer 2211 The first downsampling layer 1400 Electronic devices 2212 The second downsampling layer 1401 processor 2213 The third downsampling layer 1402 Memory DETAILED DESCRIPTION

[0068] The present application is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present application, but do not limit the present application in any form. It should be noted that, for those of ordinary skill in the art, several variations and improvements can also be made without departing from the concept of the present application. These all belong to the protection scope of the present application.

[0069] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0070] It should be noted that, if there is no conflict, the various features in the embodiments of the present application can be combined with each other, all within the scope of protection of the present application. In addition, although the functional module division is performed in the device schematic diagram and the logical order is shown in the flow chart, in some cases, the steps shown or described can be performed in a sequence different from the module division in the device or the flow chart. In addition, the words "first", "second", "third", etc. used herein do not limit the data and execution order, but only distinguish the same items or similar items with basically the same functions and effects.

[0071] Unless otherwise defined, all technical and scientific terms used in this specification have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used in this specification and in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application. The term "and / or" used in this specification includes any and all combinations of one or more of the related listed items.

[0072] In addition, the technical features involved in each embodiment of the present application described below can be combined with each other as long as they do not conflict with each other.

[0073] Before explaining the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations:

[0074] (1) Neural network, also referred to as neural network (NNs) or connection model, is an algorithmic mathematical model that imitates the behavioral characteristics of animal neural networks and performs distributed parallel information processing. Neural networks rely on the complexity of the system to adjust the interconnected relationships between a large number of internal nodes to achieve the purpose of processing information. Specifically, a neural network can be composed of neural units, which can be specifically understood as a neural network with an input layer, a hidden layer, and an output layer. Generally speaking, the first layer is the input layer, the last layer is the output layer, and the layers in between are all hidden layers. Among them, a neural network with many hidden layers is called a deep neural network (DNN). The work of each layer in a neural network can be described by the mathematical expression y=a(W·x+b). From a physical level, the work of each layer in a neural network can be understood as completing the transformation from input space to output space (i.e., the row space to the column space of a matrix) through five operations on the input space (a set of input vectors). These five operations include: 1. Dimension increase / reduction; 2. Enlargement / reduction; 3. Rotation; 4. Translation; 5. "Bending". The operations of 1, 2, and 3 are completed by "W·x", the operation of 4 is completed by "+b", and the operation of 5 is implemented by "a()". The word "space" is used here because the classified object is not a single thing, but a class of things. Space refers to the collection of all individuals of this class of things. W is the weight matrix of each layer of the neural network. Each value in the matrix represents the weight value of a neuron in this layer. The matrix W determines the spatial transformation from the input space to the output space mentioned above, that is, the W of each layer of the neural network controls how to transform the space. The purpose of training a neural network is to finally obtain the weight matrix of all layers of the trained neural network. Therefore, the training process of a neural network is essentially to learn how to control spatial transformation, and more specifically to learn the weight matrix.

[0075] It should be noted that in the embodiments of the present application, the models used in the machine learning tasks are essentially neural networks. Common components in neural networks include convolutional layers, activation function layers, and batch normalization layers. By assembling these common components in neural networks, a model is designed. When the model parameters (weight matrices of each layer) are determined so that the model error meets the preset conditions or the number of model parameters is adjusted to reach a preset threshold, the model converges.

[0076] The convolution layer is configured with multiple convolution kernels, each of which is set with a corresponding step size to perform convolution operations on the image. The purpose of the convolution operation is to extract different features of the input image. The first convolution layer may only extract some low-level features such as edges, lines, and corners. Deeper convolution layers can iteratively extract more complex features from low-level features.

[0077] The activation function layer is used to make each neuron in the neural network accept the output value of the previous layer as the input value of the neuron, and pass the processing result to the next layer. Common activation functions of the activation function layer include but are not limited to the Rectified Linear Unit (Relu) function, Swish function, and Parametric Rectified Linear Unit (PRelu) function with parameters.

[0078] Batch Normalization (BN) layer is used to standardize the features of a certain layer in the network. Its purpose is to solve the problem of numerical instability in deep neural networks, that is, to solve the problem that as the number of network layers increases, the update of parameters during training is likely to cause drastic changes in the feature output close to the output layer, which is not conducive to training an effective neural network.

[0079] (2) Loss function refers to a function that maps the value of a random event or its related random variables to a non-negative real number to represent the "risk" or "loss" of the random event. The loss function is a non-negative real function that is used to quantify the difference between the predicted label and the true label predicted by the model. In applications, the loss function is usually associated with the optimization problem as a learning criterion, that is, the model is solved and evaluated by minimizing the loss function. For example, it is used for model parameter estimation in statistics and machine learning. In the process of training a neural network, because we hope that the output of the neural network is as close as possible to the value we really want to predict, we can compare the current network's predicted value with the desired target value, and then update the weight matrix of each layer of the neural network according to the difference between the two (there is usually an initialization process before the first update, that is, pre-configuring parameters for each layer in the neural network). For example, if the network's predicted value is too high, adjust the weight matrix to make it predict a lower value, and continue to adjust until the neural network can predict the desired target value. Therefore, it is necessary to predefine "how to compare the difference between the predicted value and the target value", which is the loss function or objective function, which is an important equation used to measure the difference between the predicted value and the target value. Taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference, so the training of the neural network becomes a process of minimizing this loss as much as possible.

[0080] The technical solution of the present application is described in detail below in conjunction with the accompanying drawings.

[0081] See also Figure 1 , Figure 1 It is a schematic diagram of an application environment of a training method for an aging image generation model provided in an embodiment of the present application;

[0082] like Figure 1 As shown, the application environment 100 includes: a terminal 101 and a server 102, and the terminal 101 and the server 102 communicate through wired or wireless communication.

[0083] The terminal 101 may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited thereto. The terminal 101 may be provided with a client, which may be a video client, a browser client, an online shopping client, an instant messaging client, etc., and this application does not limit the type of the client.

[0084] The terminal 101 and the server 102 may be directly or indirectly connected via wired or wireless communication, which is not limited in this application. The terminal 101 may receive facial images of different age values ​​and text information corresponding to the facial images sent by the server 102, and display the facial images of different age values ​​and the aging images corresponding to the facial images of different age values ​​on a visual interface. The terminal may respond to an aging image generation instruction triggered by a user, and acquire facial images of different age values ​​through an image acquisition device, which may be built into the terminal 101 or externally connected to the terminal 101, which is not limited in this application.

[0085] Terminal 101 can send the aging image generation instruction and the collected facial images of different age values ​​and text information corresponding to the facial images to server 102, and receive the aging image corresponding to the facial image returned by server 102, and then display the aging image corresponding to the facial image on a visual interface so that the user can view the aging image corresponding to the facial image.

[0086] It is understandable that the terminal 101 may generally refer to one of a plurality of terminals, and the embodiment of the present application is only illustrated by the terminal 101. Those skilled in the art may know that the number of the above terminals may be more or less. For example, the above terminal may be only one, or the above terminals may be dozens or hundreds, or more. The embodiment of the present application does not limit the number of terminals and device types.

[0087] Among them, server 102 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), as well as big data and artificial intelligence platforms.

[0088] The server 102 and the terminal 101 may be directly or indirectly connected via wired or wireless communication, which is not limited in this application. The server 102 may maintain a clothing image database for storing multiple clothing images. The server 102 may receive an aging image generation instruction and facial images of different age values ​​and text information corresponding to the facial images sent by the terminal 101, and perform image-text matching on the facial image and the text information corresponding to the facial image according to the aging image generation instruction, obtain an aging image corresponding to the facial image, and then send the aging image corresponding to the facial image to the terminal 101.

[0089] It is understandable that the number of the above servers 102 may be more or less, and the embodiment of the present application does not limit this. Of course, the server 102 may also include other functional servers to provide more comprehensive and diversified services.

[0090] The technical solution of the present application can be applied to various scenarios of aging image generation. Specifically, the technical solution of the present application can be used to match text and images between a face image and text information corresponding to the face image in the scenario of aging image generation. In the scenario of aging image generation, in some implementations, the face image of the scene and the text information corresponding to the face image are fused by an aging image generation model to determine the aging image corresponding to the face image, wherein the aging image generation model is pre-trained.

[0091] Currently, large models tend to extract universal features rather than individual differences when dealing with complex aging changes, resulting in a lack of specificity in the generated aging images, which in turn leads to little difference in the aging features learned by the model.

[0092] In view of this, the embodiments of the present application provide a training method, a generation method and an electronic device for an aging image generation model to enhance the generation effect of aging images.

[0093] Embodiment 1

[0094] See also Figure 2 , Figure 2 is a structural schematic diagram of an aging image generation model provided in an embodiment of the present application;

[0095] like Figure 2 As shown, the aging image generation model 200 includes a multimodal pre-training network 201 and an aging generation network 202.

[0096] Among them, the multimodal pre-training network 201 (Contrastive Language-Image Pre-training, CLIP) is a multimodal pre-training model for contrastive learning. The basic principle of CLIP is contrastive learning, that is, letting the model learn to distinguish positive samples (matching image and text pairs) and negative samples (unmatched image and text pairs). CLIP adopts a multimodal encoder, which is composed of a multimodal text encoding network 211 and a multimodal image encoding network 212. The multimodal text encoding network 211 includes a text editor, and the multimodal image encoding network 212 includes an image editor. The multimodal text encoding network 211 is a Transformer-based model, and the multimodal image encoding network 212 can be a convolutional neural network (CNN)-based model or a visual transformer (ViT) model. The multimodal text encoding network 211 and the multimodal image encoding network 212 are both used to convert input data into a fixed-length vector representation.

[0097] The aging generation network 202 is used to match the face image with the text information corresponding to the face. The text information and its corresponding text encoding features are input into the aging generation network 202 to obtain a face image feature map.

[0098] See also Figure 3 , Figure 3 It is a flowchart of a training method for an aging image generation model provided in an embodiment of the present application;

[0099] The method can be applied to various electronic devices, such as terminals, servers and other electronic devices. Specifically, the execution subject of the method is one or at least two processors of the electronic device.

[0100] like Figure 3 As shown, the training method of the aging image generation model includes:

[0101] Step S301: Acquire a training data set, wherein the training data set includes face images of different age values ​​and text information corresponding to the face images;

[0102] It is understandable that before training the aging image generation model, it is necessary to collect a training data set and perform model training based on the training data set.

[0103] Specifically, the training data set includes face images of different age values ​​and text information corresponding to the face images, wherein the age value ranges from 1 to 100 years old, and the text information includes the real age value and the predicted age value of the face image.

[0104] It is understandable that since there may be problems with different image sizes and the front side of the image not facing upwards in the process of collecting actual facial images, the embodiment of the present application also needs to preprocess the facial image to obtain an adjusted facial image so that the adjusted facial image has a consistent size and the front side of the adjusted facial image faces upwards.

[0105] In the embodiment of the present application, the face image is preprocessed to obtain the adjusted face image, specifically including: obtaining the coordinates of the face key points in the face image according to the face key point algorithm, obtaining the effective face area according to the face key point coordinates, and normalizing the scale of the effective face area to obtain the adjusted face image. Among them, the face key points are important feature points of various parts of the face, and the face key points include eyebrows, eyes, nose, mouth, and points in the facial contour area.

[0106] Among them, the face key point algorithm uses 106 points to mark the face key points, and determines the effective area of ​​the face based on the 106 key points obtained, among which the 105th and 106th key points are the coordinate points of the eyeball center.

[0107] Please refer to Figure 4 , Figure 4 is a schematic diagram of key points of a face provided in an embodiment of the present application;

[0108] like Figure 4 As shown, the 106 key points of the face are obtained using the 106-point labeled face key point algorithm, wherein the 106 key points include 33 key points of the outer contour, 15 key points of the nose, 20 key points of the mouth, 20 key points of the eyes, 18 key points of the eyebrows, and the 105th and 106th points are the center points of the eyeballs.

[0109] In the embodiment of the present application, after obtaining the eyeball center coordinate point, the angle θ of the left-right rotation of the face is calculated according to the eyeball center coordinate point, and the tilted face image is adjusted through the rotation change matrix, including:

[0110]

[0111] Among them, (x, y) is the two-dimensional coordinate corresponding to the pixel position in the original image, (x ′ ,y ′ ) is the adjusted two-dimensional coordinate corresponding to the pixel position in the original image.

[0112] In the embodiment of the present application, the tilted face image is adjusted by rotating the transformation matrix so that the front side of the face image faces upward.

[0113] Furthermore, after adjusting all tilted face images, scale normalization is performed to unify the size of all face images, for example, the size of the face images is unified to 1024*1024. Alternatively, operations such as illumination transformation can be performed on each face image for data enhancement, which helps to improve the robustness of the aging image generation model.

[0114] Step S302: inputting the text information into the multimodal pre-training network to obtain text encoding features corresponding to the text information;

[0115] Specifically, the text information corresponding to the face image is input into the multimodal text encoding network of the multimodal pre-trained network, and the text information is encoded by the multimodal text encoding network to obtain the text encoding features corresponding to the text information. The text information includes the real age value and the predicted age value of the face image. For example, the real age value of the input face image is a years old, and the age value of the generated image is b years old, where b is the predicted age value, and a is the predicted age value. <b。

[0116] In the embodiment of the present application, the predicted age value can be generated using a random number generator, and the range of the predicted age value is 0-100 years old.

[0117] Step S303: inputting the face image and text encoding features into the aging generation network to obtain a face image feature map;

[0118] Specifically, the text encoding features corresponding to the text information are input into the aging generation network to obtain a facial image feature map.

[0119] Please refer to Figure 5 , Figure 5 It is a schematic diagram of the structure of an aging generation network provided in an embodiment of the present application;

[0120] like Figure 5 As shown, the aging generation network 202 includes a downsampling module 2021, an intermediate module 2022, and an upsampling module 2023.

[0121] The down-sampling module 2021 is used to down-sample the face image to extract image features of the face image.

[0122] The intermediate module 2022 is used to perform feature fusion on the feature map and text encoding features output by the downsampling module.

[0123] The upsampling module 2023 is used to perform feature fusion on the text encoding features and the feature map, and to upsample the feature map at the same time.

[0124] In an embodiment of the present application, a U-shaped structure is formed by a downsampling module, an intermediate module, and an upsampling module. The U-shaped structure is used to learn and fuse image features and text encoding features to enhance the generation effect of aging change features.

[0125] Please refer to Figure 6 , Figure 6 yes Figure 3 A detailed flowchart of step S303 in FIG.

[0126] like Figure 6 As shown, step S303 includes:

[0127] Step S3301: inputting the face image into a downsampling module for feature extraction to obtain a first image feature map;

[0128] In an embodiment of the present application, the aging generation network also includes a first convolution module. Before the face image is input into the downsampling module, the number of channels of the face image is increased by the first convolution module to extract more features of the face image. For example, the number of channels of the currently input face image is 3. After being processed by the first convolution module, the number of channels of the face image is increased to 64. By adding multiple channels, the size of the original image can be kept unchanged and more features of the face image can be extracted in different dimensions, so as to fuse the features of multiple channels and output a 64-channel face image.

[0129] Specifically, the face image output by the first convolution module is input into the downsampling module for feature extraction to obtain a first image feature map.

[0130] Please refer to Figure 7 , Figure 7 is a structural diagram of a downsampling module provided in an embodiment of the present application;

[0131] like Figure 7As shown, the downsampling module 2021 includes at least four sequentially connected downsampling layers, namely a first downsampling layer 2211, a second downsampling layer 2212, a third downsampling layer 2213, and a fourth downsampling layer 2214. Each downsampling layer includes an instance normalization layer, an activation function layer, a convolution layer, and an average pooling layer. The instance normalization layer normalizes the extracted features so that the feature distribution of each sample (i.e., the face image) is adjusted to a uniform scale to help the model learn the differences between samples. The activation function layer is used to enable the model to extract more nonlinear features. Features are used to enhance the expressiveness of the model. The number of convolution kernels of each downsampling layer is set to 128, 256, 512, and 512 respectively. The face image is extracted through the convolution layer. The step size of the average pooling layer is 2. After each downsampling layer, the size of the face image is reduced to twice its original size. The extracted features include the edge, contour, color, texture, facial structure, local texture pattern, overall face, and age-related features of the face. The face image is further downsampled through the average pooling layer to reduce the size of the face image and passed.

[0132] In an embodiment of the present application, the face image is downsampled by at least four sequentially connected downsampling layers in a downsampling module to output at least four first image feature maps of different scales. For example, if the current input face image is 1024*1024*64, then after being processed by the downsampling module, first image feature maps of 512*512*128, 256*256*256, 128*128*512, and 64*64*512 are output respectively.

[0133] Step S3302: inputting the first image feature map and the text encoding features into the intermediate module for feature fusion to obtain a second image feature map;

[0134] Specifically, the first image feature map and the text encoding features are input into the intermediate module for feature fusion to obtain the second image feature map.

[0135] Please refer to Figure 8 , Figure 8 It is a structural schematic diagram of an intermediate module provided in an embodiment of the present application;

[0136] like Figure 8As shown, the intermediate module 2022 includes a first intermediate layer 2221 and a second intermediate layer 2222. The first intermediate layer is used to perform feature extraction on the first image feature map to extract more detailed features of the facial image, such as skin texture, wrinkle depth and distribution, etc. The second intermediate layer is used to perform preliminary feature fusion of the extracted facial image features and text encoding features, which helps the model understand the relationship between text information and image features, so that the generated second image feature map meets the requirements of text description.

[0137] Please refer to Fig. 9 , Fig. 9 yes Figure 6 A detailed flowchart of step S3302 in FIG.

[0138] like Fig. 9 As shown, step S3302 includes:

[0139] Step S3321: input the first image feature map into the first intermediate layer for feature extraction to obtain a high-level semantic feature map;

[0140] Specifically, the first image feature map is input into the first intermediate layer, and the first intermediate layer performs feature extraction on the first image feature map to obtain a high-level semantic feature map, wherein the high-level semantic feature map includes high-level features, which are more abstract and general than low-level features (such as edges, textures, colors, etc.), and they focus more on the overall structure and semantic content of the image, such as the overall contour of the face, the layout of the facial features, expression features, and facial details (wrinkles, sagging skin, etc.).

[0141] Among them, the first intermediate layer includes at least two feature extraction layers connected in sequence, each feature extraction layer includes an instance normalization layer, an activation function layer, and a convolution layer. The instance normalization layer is used to normalize the extracted high-level semantic features so that the feature distribution of each sample is adjusted to a uniform scale. The activation function layer is used to enable the model to extract more nonlinear features to enhance the expressive ability of the model. The number of convolution kernels of the convolution layer of each feature extraction layer is set to 512 and 512 respectively. The convolution layer is used to perform high-level feature extraction on the first image feature map to obtain high-level semantic features.

[0142] In an embodiment of the present application, by performing feature extraction on the first image feature map in the first intermediate layer to obtain a high-level semantic feature map, image features of deeper dimensions can be extracted, so that the model can more accurately understand the overall structure of the facial image, so as to further generate a more natural and realistic aging image.

[0143] Step S3322: inputting the high-level semantic feature map and the text encoding feature into the second intermediate layer for feature fusion to obtain a second image feature map;

[0144] Specifically, the high-level semantic feature map and the text encoding feature are input into the second intermediate layer, and the high-level semantic feature map and the text encoding feature are associated through the second intermediate layer to fuse the feature map with the features of the text information to obtain the second image feature map.

[0145] The second intermediate layer includes at least two feature fusion layers connected in sequence, each feature fusion layer includes an adaptive normalization layer, an activation function layer, and a convolution layer, the convolution layer is used to extract image features of a high-level semantic feature map, and the adaptive normalization layer is used to perform feature fusion on the high-level semantic feature map and the text encoding feature to obtain a second image feature map, including:

[0146]

[0147] Among them, F is the text encoding feature, x is the high-level semantic feature map, σ(F) is the mean of the text encoding feature, σ(x) is the mean of the high-level semantic feature map, μ(x) is the variance value of the high-level semantic feature map, and μ(F) is the variance value of the text encoding feature.

[0148] Among them, the Adaptive Layer Normalization (AdaLN) introduces an adaptive mechanism based on layer normalization. The adaptive mechanism dynamically generates normalization parameters (such as the mean σ(F) of text encoding features and the variance μ(F) of text encoding features) according to input conditions (such as text encoding features), and obtains the mapping relationship from input conditions to normalization parameters through neural network learning. Traditional normalization layers, such as instance normalization layers, use a fixed number of normalization layers. By introducing an adaptive mechanism in the normalization layer, modal information such as text can be effectively integrated into image features, enhancing the expressiveness of features and significantly improving the performance and adaptability of the model.

[0149] In an embodiment of the present application, the feature map is fused with the text encoding feature through the above formula, and the distribution of image features can be dynamically adjusted according to the text encoding feature, so that the model can adapt more flexibly when facing different text and image combinations, thereby improving the generalization ability and robustness of the model.

[0150] Step S3303: inputting the second image feature map and the text encoding features into an upsampling module to obtain a face image feature map;

[0151] Specifically, the second image feature map and the text encoding features are input into an upsampling module, and the upsampling module performs feature extraction and fusion on the second image feature map and the text encoding features to obtain a facial image feature map.

[0152] At the same time, according to the characteristics of the U-shaped network structure, the first image feature map with the same number of channels as the second image feature map is input into the upsampling module, so that the upsampling module simultaneously integrates the underlying features, high-level features and text encoding features, which helps the face image feature map generated by the model to be more refined and realistic.

[0153] Please refer to Fig.10 , Fig.10 is a structural diagram of an upsampling module provided in an embodiment of the present application;

[0154] like Fig.10 As shown, the upsampling module 2023 includes at least four upsampling layers connected in sequence, namely a first upsampling layer 2231, a second upsampling layer 2232, a third upsampling layer 2233, and a fourth upsampling layer 2234. Each upsampling layer includes an adaptive normalization layer, an activation function layer, a convolution layer, and an upper pooling layer. The number of convolution kernels of the convolution layers of the four upsampling layers in the upsampling module is set to 512, 256, 128, and 64, respectively. By setting convolution layers with different numbers of channels, feature extraction is performed on the first image feature map and the second image feature map to obtain more image features. The step size of the upper pooling layer is set to 2. After each layer of upper pooling layer processing, the size of the feature map will double. The extracted image features and text encoding features are normalized by the adaptive normalization layer to achieve mutual complementation and enhancement of image and text information, thereby improving the performance of the model in cross-modal tasks. Among them, the facial image feature map output by the previous upsampling layer is also used as the input data of the next upsampling layer, so that the aging change characteristics in different facial image feature maps generated by the model are different.

[0155] In an embodiment of the present application, feature fusion is performed on the image feature map and the text encoding feature through at least four sequentially connected upsampling layers in the upsampling module to output at least four facial image feature maps of different scales, for example, facial image feature maps of 128*128*512, 256*256*256, 512*512*128, and 1024*1024*64 are output respectively.

[0156] In an embodiment of the present application, the fusion of the image feature map and the text encoding feature in the upsampling module adopts the formula corresponding to generating the second image feature map in the above step S3322.

[0157] In an embodiment of the present application, the aging generation network also includes a second convolution module, which is used to reduce the number of channels of the face image feature map output by the upsampling module so that the model focuses on local detail features to output high-quality face image feature maps. For example, a 64-channel face image feature map is reduced to a 3-channel face image feature map, and multi-channel image features can be fused to extract higher-level features, so that the retained channels can better improve feature expression capabilities.

[0158] Please refer to Table 1 again, which is the overall structure of an aging generation network provided in an embodiment of the present application;

[0159]

[0160]

[0161] Table 1

[0162] As shown in Table 1, the aging generation network includes a first convolution module, a downsampling module, an intermediate module, an upsampling module, and a second convolution module. The downsampling module includes a first downsampling layer, a second downsampling layer, a third downsampling layer, and a fourth downsampling layer. The intermediate module includes a first intermediate layer, a second intermediate layer, and the first intermediate layer includes a first feature extraction layer and a second feature extraction layer. The second intermediate layer includes a first feature fusion layer and a second feature fusion layer. The upsampling module includes a first upsampling layer, a second upsampling layer, a third upsampling layer, and a fourth upsampling layer.

[0163] Among them, the first convolution module is used to increase the number of channels for feature extraction of the input face image. For example, if a face image of 1024*1024*3 is input, after being processed by the first convolution module, a face image of 1024*1024*64 is output, that is, a feature map of 64 channels is output.

[0164] Among them, each downsampling layer in the downsampling module includes an average pooling layer, a convolution layer, and an instance normalization layer. The average pooling layer is used to perform an average pooling operation on the feature map, so that the size of the feature map is reduced in turn after being processed by each downsampling layer, and four feature maps of different sizes are output. For example, a 1024*1024 face image is input to the downsampling module, and after being processed in turn by each layer of the downsampling module, feature maps of 512*512, 256*256, 128*128, and 64*64 are output respectively. The number of convolution kernels of the convolution layer of each downsampling layer in the downsampling module is set to 128, 256, 512, and 512, and a feature map of 512 channels is output after multiple layers of convolution. After each downsampling layer extracts features, the extracted image features are normalized using instance normalization, that is, by calculating the mean and variance of each sample (i.e., face image) on each feature channel, all elements of the channel are normalized using the mean and variance.

[0165] Among them, each feature extraction layer of the first intermediate layer of the intermediate module includes an instance normalization layer and a convolution layer, and the extracted image features are normalized by the instance normalization layer. Each feature fusion layer in the second intermediate layer of the intermediate module includes an adaptive normalization layer and a convolution layer, and the extracted image features and text encoding features are normalized by the adaptive normalization layer to fuse the image features and text encoding features. The adaptive normalization layer introduces an adaptive mechanism to adaptively generate normalization parameters according to the input data, so that the model can better generalize to different data distributions to improve the generalization ability and robustness of the model. The number of convolution kernels of the convolution layer in each intermediate layer in the intermediate module is set to 512, 512, 512, 512, so that the model can better extract and integrate features in the specified direction.

[0166] Among them, each upsampling layer in the upsampling module includes an up-pooling layer, a convolution layer, and an adaptive normalization layer. The up-pooling layer is used to perform an upsampling operation on the feature map, so that the size of the feature map is increased in turn after being processed by each upsampling layer, and four feature maps of different sizes are output. For example, a 64*64 face image is input to the downsampling module, and after being processed in turn by each layer of the upsampling module, feature maps of 128*128, 256*256, 512*512, and 1024*1024 are output respectively. The number of convolution kernels of the convolution layer of each upsampling layer in the upsampling module is set to 512, 256, 128, and 64, and a feature map of 64 channels is output after multiple layers of convolution. After each upsampling layer extracts features, the extracted image features are normalized using an adaptive normalization method to adaptively generate normalization parameters according to the input data, and the feature map and text encoding features are normalized according to the normalization parameters.

[0167] Among them, the second convolution module is used to condense the number of channels of the input feature map. For example, if a 1024*1024*64 feature map is input, after being processed by the second convolution module, a 1024*1024*3 face image feature map is output, that is, a 3-channel feature map is output.

[0168] In an embodiment of the present application, a face image is input into an aging generation network, the number of channels corresponding to the input face image is increased through a first convolution module to extract image features of the face image in multiple dimensions, the face image output by the first convolution module is input into a downsampling module, the face image is downsampled through multiple downsampling layers in the downsampling module to obtain a first image feature map, further, the first image feature map and the text encoding features are feature fused through an intermediate module to obtain a second image feature map, the second image feature map and the text encoding features are upsampled through an upsampling module to output a face image feature map, and finally the number of channels of the face image feature map is condensed through a second convolution module, so that the model focuses on local detail features and outputs a high-quality face image feature map.

[0169] Step S304: inputting the facial image feature map into the multimodal pre-training network to obtain an aging image;

[0170] Specifically, the facial image feature map is input into the multimodal pre-trained network, and the image information feature of the facial image feature map is encoded by the image editor of the multimodal pre-trained network to obtain an aging image.

[0171] Step S305: constructing a model loss function, training the aging image generation model based on the training data set until the model loss function converges, so as to generate a trained aging image generation model, wherein the model loss function includes a similarity loss;

[0172] Specifically, a model loss function is constructed, and the aging image generation model is trained based on the training data set until the model loss function converges to generate the trained aging image generation model, wherein the model loss function includes a similarity loss, and the training data set includes face images of different age values ​​and text information corresponding to the face images.

[0173] Specifically, the similarity loss includes:

[0174]

[0175] Among them, Loss is the similarity loss, L 正 is the similarity loss between the positive text feature and the face image, L 反 is the similarity loss between the inverse text features and the face image.

[0176] Among them, Loss uses the Softmax function, which is used to convert any real number K-dimensional vector z into another K-dimensional vector sigma, where the range of each element is between (0, 1] and the sum of all elements is 1. The Softmax function is applied to multi-classification problems, and its goal is to maximize the model's prediction probability for the correct category, that is, to minimize the difference between the predicted probability and the true label.

[0177] In an embodiment of the present application, in order to eliminate the influence of language ambiguity on the accuracy of image-text matching, forward text information and reverse text information are obtained, and the similarity losses of the forward text information, reverse text information and facial images are constructed respectively, so as to guide the model to pay more attention to the consistency between text and image features during the training process, thereby optimizing the matching performance of the model. For example, the forward text is: the current input facial image is a years old, and the generated facial image is b years old; the reverse text is: the current input facial image is a years old, and the generated facial image is not b years old.

[0178] Specifically, the similarity loss between the positive text and the face image includes:

[0179]

[0180] Among them, L 正 is the similarity loss between the positive text and the face image, F 1 is the positive text feature, F I A face image.

[0181] Specifically, the similarity loss between reverse text and face images includes:

[0182]

[0183] Among them, L 反 is the similarity loss between the positive text and the face image, F′ 1 is the reverse text feature, F I A face image.

[0184] In the embodiment of the present application, by separately constructing the similarity loss between the forward text and the face image and the similarity loss between the reverse text and the face image, the deviation in model understanding caused by language ambiguity can be reduced, which helps to improve the accuracy of image-text matching.

[0185] In an embodiment of the present application, a model loss function is used for back propagation so that the aging generation network in the aging image generation model converges, wherein the network convergence includes the loss reaching a minimum and / or the loss fluctuating within a range and / or the number of training times reaching a certain number. For example: the Adam algorithm (Adaptive Moment Estimation Algorithm) is used to optimize the network parameters of the aging generation network in the aging image generation model, the number of iterations is set to 10,000 times, the initialization learning rate is set to 0.001, the weight decay is set to 0.0005, and the learning rate decays to 1 / 10 of the original every 1,000 iterations. The network is trained until convergence to save the corresponding network parameters at the time of convergence. Among them, the model training can be carried out using a graphics card, for example: dual-card parallel training, and the number of samples selected for one training (batchsize) can be set according to specific needs.

[0186] It can be understood that the Adam algorithm can be seen as a combination of the momentum method and the RMSprop algorithm. It not only uses momentum as the parameter update direction, but also can adaptively adjust the learning rate.

[0187] By constructing a model loss function, which includes similarity loss, and using the model loss function to train the aging image generation model, the matching degree of the aging image generation model for text information and facial images can be improved, so that the aging image generation model can learn more aging change characteristics and generate aging images with great differences based on different facial images, thereby enhancing the aging image generation effect of the aging image generation model.

[0188] Please refer to Fig.11 , Fig.11 is a schematic diagram of generating an aging image provided by an embodiment of the present application;

[0189] like Fig.11 As shown, text information is obtained, and the text information includes forward text and reverse text. The forward text and reverse text are feature encoded by a text editor in a multimodal pre-trained network to obtain forward text features and reverse text features. The face image, forward text features, and reverse text features are further input into an aging generation network. The face image is feature matched with the forward text features and reverse text features by the aging generation network to output a face image feature map. Finally, the face image feature map is encoded with image information by an image editor in the multimodal pre-trained network to obtain an aging image.

[0190] Please refer to Fig.12 , Fig.12 It is a schematic diagram of an overall process of generating an aging image provided by an embodiment of the present application;

[0191] like Fig.12 As shown in FIG. 1 , the overall process of generating aging images includes:

[0192] Step S1201: Acquire a training data set, wherein the training data set includes face images of different age values ​​and text information corresponding to the face images;

[0193] Step S1202: feature encoding the text information;

[0194] Specifically, feature encoding is performed on text information corresponding to the face image through a text editor in a multimodal pre-trained network to obtain text encoding features.

[0195] Step S1203: performing feature fusion on the face image and text information;

[0196] Specifically, the face image and text encoding features are input into the aging generation network, and the face image and text encoding features are fused through the aging generation network to output a face image feature map.

[0197] Step S1204: constructing a model loss function to train the aging image generation model;

[0198] Specifically, a model loss function is constructed to train the aging image generation model, and the aging image generation model is trained based on the training data set until the model loss function converges to generate the trained aging image generation model.

[0199] Step S1205: outputting an aging image;

[0200] Specifically, actual face images and text information are input into the trained aging image generation model to output an aging image.

[0201] In an embodiment of the present application, the aging image generation model is trained based on the training data set, which can integrate various types of feature information, so that the model can better handle complex and changeable data and improve the generalization ability of the model.

[0202] In an embodiment of the present application, a training method for an aging image generation model is provided, the aging image generation model includes: a multimodal pre-trained network, an aging generation network, and the method includes: obtaining a training data set, wherein the training data set includes face images of different age values ​​and text information corresponding to the face images; inputting the text information into the multimodal pre-trained network to obtain text encoding features corresponding to the text information; inputting the face image and the text encoding features into the aging generation network to obtain a face image feature map; inputting the face image feature map into the multimodal pre-trained network to obtain an aging image; constructing a model loss function, and training the aging image generation model based on the training data set until the model loss function converges to generate a trained aging image generation model, wherein the model loss function includes a similarity loss.

[0203] On the one hand, by obtaining face images of different age values ​​and text information corresponding to the face images, inputting the text information into a multimodal pre-trained network to obtain text encoding features, inputting the face images and text encoding features into an aging generation network to obtain a face image feature map, and inputting the face image feature map into a multimodal pre-trained network to obtain an aging image, the present application can improve the generation capability of an aging image generation model, so that the model can learn more aging change features to adapt to the differences in aging changes of different faces, and enhance the image generation capability of the aging image generation model.

[0204] On the other hand, by constructing a model loss function for model training, the model loss function includes a similarity loss to obtain a trained aging image generation model. By training with similarity loss, the present application can enhance the generation effect of aging images.

[0205] Embodiment 2

[0206] After the training method of the aging image generation model provided in the above embodiment, a trained aging image generation model is obtained, and the aging image generation model can be used to generate actual aging images.

[0207] It can be understood that the above-mentioned embodiment 1 is the training stage of the aging image generation model, and the embodiment 2 of the present application is the use stage of the aging image generation model to obtain the aging image.

[0208] For details, please refer to Fig.13 , Fig.13 It is a flowchart of a method for generating an aging image provided in an embodiment of the present application;

[0209] like Fig.13 As shown, the aging image generation method comprises:

[0210] Step S1301: Acquire a face image and text information, wherein the text information includes a real age value and a predicted age value of the face image;

[0211] Specifically, after obtaining the trained aging image generation model, a face image and text information corresponding to the face image are obtained, wherein the text information includes a real age value and a predicted age value of the face image.

[0212] The aging image generation model is trained by the composite neural network (including the multimodal pre-trained network and the aging generation network) in the above-mentioned embodiment, and has the same structure and function as the above-mentioned composite neural network. The specific processing method of the aging image generation model for facial images and text information is the same as the processing method of the composite neural network for facial images and text information in the above-mentioned embodiment, and will not be repeated here.

[0213] Step S1302: inputting the face image and text information into the aging image generation model to obtain an aging image corresponding to the face image.

[0214] It should be noted that the aging image generation model in the embodiment of the present application is a model obtained by training in the above-mentioned embodiment 1. The relevant content of the aging image generation model can refer to the content mentioned in the above-mentioned embodiment and will not be repeated here.

[0215] In an embodiment of the present application, an aging image generation model is trained, and after the trained aging image generation model is obtained, a facial image and text information are obtained, and the facial image and text information are input into the aging image generation model to obtain an aging image corresponding to the face, and the age value corresponding to the generated aging image is the predicted age value. The present application can improve the generation effect of aging images.

[0216] Embodiment 3

[0217] See also Fig.14 , Fig.14 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application;

[0218] like Fig.14 As shown, the electronic device 1400 includes one or more processors 1401 and a memory 1402. Fig.14 A processor 1401 is taken as an example.

[0219] The processor 1401 and the memory 1402 may be connected via a bus or other means. Fig.14 The example of connecting through bus is taken in the following.

[0220] The processor 1401 is used to provide computing and control capabilities to control the electronic device 1400 to perform corresponding tasks, for example, to control the electronic device 1400 to perform the training method of the aging image generation model in any one of the above method embodiments, including:

[0221] Obtain a training data set, wherein the training data set includes face images of different age values ​​and text information corresponding to the face images; input the text information into a multimodal pre-training network to obtain text encoding features corresponding to the text information; input the face images and the text encoding features into an aging generation network to obtain a face image feature map; input the face image feature map into a multimodal pre-training network to obtain an aging image; construct a model loss function, and train an aging image generation model based on the training data set until the model loss function converges to generate an aging image generation model after training, wherein the model loss function includes a similarity loss.

[0222] On the one hand, by obtaining face images of different age values ​​and text information corresponding to the face images, inputting the text information into a multimodal pre-trained network to obtain text encoding features, inputting the face images and text encoding features into an aging generation network to obtain a face image feature map, and inputting the face image feature map into a multimodal pre-trained network to obtain an aging image, the present application can improve the generation capability of an aging image generation model, so that the model can learn more aging change features to adapt to the differences in aging changes of different faces, and enhance the image generation capability of the aging image generation model.

[0223] On the other hand, by constructing a model loss function for model training, the model loss function includes a similarity loss to obtain a trained aging image generation model. By training with similarity loss, the present application can enhance the generation effect of aging images.

[0224] Alternatively, controlling the electronic device 1400 to execute the aging image generation method in any one of the above method embodiments includes:

[0225] Obtain a facial image and text information, wherein the text information includes a real age value and a predicted age value of the facial image; input the facial image and text information into an aging image generation model to obtain an aging image corresponding to the facial image, wherein the aging image generation model is trained based on the method of Example 1.

[0226] By acquiring a face image and text information, and inputting the face image and text information into an aging image generation model, an aging image corresponding to the face is obtained. This application can improve the generation effect of the aging image.

[0227] The processor 1401 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), a hardware chip or any combination thereof; it may also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a programmable logic device (PLD) or a combination thereof. The above-mentioned PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof.

[0228] The memory 1402 is a non-transitory computer-readable storage medium, which can be used to store non-transitory software programs, non-transitory computer executable programs and modules, such as the training method of the aging image generation model or the program instructions / modules corresponding to the aging image generation method in the embodiment of the present application. The processor 1401 can implement the training method of the aging image generation model or the aging image generation method in any of the following method embodiments by running the non-transitory software programs, instructions and modules stored in the memory 1402. Specifically, the memory 1402 may include a volatile memory (VM), such as a random access memory (RAM); the memory 1402 may also include a non-volatile memory (NVM), such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD) or other non-transitory solid-state storage device; the memory 1402 may also include a combination of the above-mentioned types of memory.

[0229] The memory 1402 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage devices. In some embodiments, the memory 1402 may optionally include a memory remotely arranged relative to the processor 1401, and these remote memories may be connected to the processor 1401 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0230] One or more modules are stored in the memory 1402, and when executed by one or more processors 1401, the training method of the aging image generation model or the aging image generation method in any of the above method embodiments is executed, for example, the above described Figure 3 , 13 The steps shown.

[0231] In the embodiment of the present application, the electronic device 1400 may also have components such as a wired or wireless network interface, a keyboard, and an input / output interface for input and output. The electronic device 1400 may also include other components for realizing device functions, which will not be elaborated here.

[0232] The electronic device of the embodiment of the present application exists in various forms. Figure 3 , Fig.13 The various steps shown include but are not limited to: servers, mobile terminals, electronic watches, fixed terminals, wearable devices and other devices.

[0233] The present application also provides a non-volatile computer-readable storage medium, such as a memory including a program code, and the program code can be executed by a processor to complete the training method of the aging image generation model or the aging image generation method in the above embodiment. For example, the non-volatile computer-readable storage medium can be a read-only memory (ROM), a random access memory (RAM), a compact disc (CDROM), a magnetic tape, a floppy disk, an optical data storage device, etc.

[0234] The present application also provides a computer program product, which includes one or more program codes, and the program codes are stored in a non-volatile computer-readable storage medium. The processor of the electronic device reads the program code from the non-volatile computer-readable storage medium, and the processor executes the program code to complete the method steps of the training method of the aging image generation model or the aging image generation method provided in the above embodiment.

[0235] A person skilled in the art will appreciate that all or part of the steps for implementing the above embodiments may be accomplished by hardware or by hardware associated with a program code, and the program may be stored in a non-volatile computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0236] Through the description of the above implementation methods, ordinary technicians in this field can clearly understand that each implementation method can be implemented by means of software plus a general hardware platform, and of course, it can also be implemented by hardware. Ordinary technicians in this field can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0237] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Under the concept of the present application, the technical features in the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other changes in different aspects of the present application as mentioned above, which are not provided in detail for the sake of simplicity. Although the present application has been described in detail with reference to the aforementioned embodiments, a person of ordinary skill in the art should understand that the technical solutions described in the aforementioned embodiments can still be modified, or some of the technical features can be replaced by equivalents. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A training method for an aging image generation model, characterized in that: The aging image generation model includes: a multimodal pre-training network, an aging generation network, and the method includes: Acquire a training data set, wherein the training data set includes face images of different age values ​​and text information corresponding to the face images; Inputting the text information into the multimodal pre-trained network to obtain text encoding features corresponding to the text information; Inputting the face image and the text encoding features into the aging generation network to obtain a face image feature map; Inputting the facial image feature map into the multimodal pre-training network to obtain an aging image; A model loss function is constructed, and the aging image generation model is trained based on the training data set until the model loss function converges to generate a trained aging image generation model, wherein the model loss function includes a similarity loss.

2. The method according to claim 1, characterized in that The multimodal pre-trained network includes a multimodal text encoding network; The step of inputting the text information into a multimodal pre-training network to obtain text encoding features corresponding to the text information includes: The text information is input into the multimodal text encoding network, and feature encoding is performed through the multimodal text encoding network to generate text encoding features corresponding to the text information, wherein the text information includes the real age value and the predicted age value of the face image.

3. The method according to claim 1, characterized in that The aging generation network includes a downsampling module, an intermediate module, and an upsampling module; The step of inputting the face image and the text encoding features into the aging generation network to obtain a face image feature map comprises: Inputting the face image into the downsampling module for feature extraction to obtain a first image feature map; Inputting the first image feature map and text encoding features into the intermediate module for feature fusion to obtain a second image feature map; The second image feature map and text encoding features are input into the upsampling module to obtain the face image feature map.

4. The method according to claim 3, characterized in that The intermediate module includes a first intermediate layer and a second intermediate layer, the first intermediate layer includes at least two feature extraction layers connected in sequence, and the second intermediate layer includes at least two feature fusion layers connected in sequence; The step of inputting the first image feature map and the text encoding feature into the intermediate module for feature fusion to obtain the second image feature map comprises: Inputting the first image feature map into the first intermediate layer for feature extraction to obtain a high-level semantic feature map; The high-level semantic feature map and text encoding features are input into the second intermediate layer for feature fusion to obtain the second image feature map.

5. The method according to claim 4, characterized in that The downsampling module includes at least four sequentially connected downsampling layers, each of which is used to perform a downsampling operation on the face image to obtain at least four first image feature maps of different scales; The upsampling module includes at least four upsampling layers connected in sequence, and each upsampling layer is used to perform an upsampling operation on the second image feature map and the text encoding feature to obtain at least four facial image feature maps of different scales.

6. The method according to claim 4, characterized in that The multimodal pre-trained network includes: a multimodal image coding network; The step of inputting the facial image feature map into the multimodal pre-training network to obtain an aging image comprises: The facial image feature map is input into the multimodal image coding network, and feature coding is performed through the multimodal image coding network to generate an aging image.

7. The method according to claim 4, characterized in that The step of inputting the high-level semantic feature map and the text encoding feature into the second intermediate layer for feature fusion to obtain the second image feature map includes: Among them, F is the text encoding feature, x is the high-level semantic feature map, σ(F) is the mean of the text encoding feature, σ(x) is the mean of the high-level semantic feature map, μ(x) is the variance value of the high-level semantic feature map, and μ(F) is the variance value of the text encoding feature.

8. The method according to claim 1, characterized in that The text encoding features include forward text features and reverse text features; The similarity loss includes: Among them, Loss is the similarity loss, L 正 is the similarity loss between the positive text feature and the face image, L 反 is the similarity loss between the inverse text features and the face image.

9. The method according to claim 8, characterized in that The similarity loss between the positive text and the face image includes: Among them, L 正 is the similarity loss between the positive text and the face image, F1 is the positive text feature, and F I is a face image; The similarity loss between the reverse text and the face image includes: Among them, L 反 is the similarity loss between the positive text and the face image, F′1 is the reverse text feature, and F I A human face image.

10. A method for generating an aging image, characterized in that: The method comprises: Acquire a face image and text information, wherein the text information includes a real age value and a predicted age value of the face image; The facial image and text information are input into an aging image generation model to obtain an aging image corresponding to the facial image, wherein the aging image generation model is trained based on the method described in any one of claims 1 to 9.

11. An electronic device, characterized in that: include: at least one processor, and a memory communicatively coupled to the at least one processor, wherein: The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the method as described in any one of claims 1 to 10.

Images