Image migration method and training method and device of image migration model

Abstract

The present disclosure provides an image migration method and a training method, device and equipment of an image migration model, relates to the field of artificial intelligence, in particular to the field of deep learning and computer vision technology, and can be applied to the scene of face image processing, face recognition and the like. The image migration method comprises: extracting a first attribute feature of a first object included in a first image; extracting a first shape feature of a first target part included in the first object according to audio data of the first image; extracting a first identity feature of a second object included in a second image; and generating a first migration image based on the first attribute feature, the first shape feature and the first identity feature, the first migration image comprising the second object with the first attribute feature and the first shape feature.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of artificial intelligence, in particular to the field of deep learning and computer vision, and can be applied to scenarios such as face image processing and face recognition. BACKGROUND

[0002] With the development of deep learning and image processing technology, image migration technology can be applied to multiple fields such as face recognition, video production, and virtual reality.

[0003] In related technologies, three-dimensional reconstruction technology is used to realize image migration. This method is complex in steps and large in calculation amount. SUMMARY

[0004] A method for reducing the calculation amount and improving the migration accuracy of image migration, a training method, device, electronic equipment, and storage medium for an image migration model are provided.

[0005] One aspect of the present disclosure provides a method for image migration, comprising: extracting a first attribute feature of a first object included in a first image; extracting a first shape feature of a first target part included in the first object according to audio data for the first image; extracting a first identity feature of a second object included in a second image; and generating a first migration image based on the first attribute feature, the first shape feature, and the first identity feature, the first migration image including the second object having the first attribute feature and the first shape feature.

[0006] Another aspect of the present disclosure provides a training method for an image migration model, wherein the image migration model includes a first encoding network, a second encoding network, a third encoding network, and a generation network, the training method comprising: extracting a second attribute feature of a third object included in a third image using the first encoding network; extracting a second shape feature of a first target part included in the third object according to audio data for the third image using the second encoding network; extracting a second identity feature of the third object included in a fourth image using the third encoding network; generating a second migration image based on the second attribute feature, the second shape feature, and the second identity feature using the generation network; the second migration image including the third object having the second attribute feature and the second shape feature; and training the image migration model based on the difference between the second migration image and the third image.

[0007] Another aspect of the present disclosure provides an image migration device, comprising: a first attribute extraction module configured to extract a first attribute feature of a first object included in a first image; a first shape extraction module configured to extract a first shape feature of a first target part included in the first object according to audio data corresponding to the first image; a first identity extraction module configured to extract a first identity feature of a second object included in a second image; and a first image generation module configured to generate a first migration image based on the first attribute feature, the first shape feature, and the first identity feature, the first migration image comprising the second object having the first attribute feature and the first shape feature.

[0008] Another aspect of the present disclosure provides a training device of an image migration model, wherein the image migration model comprises a first encoding network, a second encoding network, a third encoding network, and a generation network; the training device comprises: a second attribute extraction module configured to extract a second attribute feature of a third object included in a third image by using the first encoding network; a second shape extraction module configured to extract a second shape feature of a first target part included in the third object according to audio data corresponding to the third image by using the second encoding network; a second identity extraction module configured to extract a second identity feature of the third object included in a fourth image by using the third encoding network; a second image generation module configured to generate a second migration image based on the second attribute feature, the second shape feature, and the second identity feature by using the generation network; the second migration image comprising the third object having the second attribute feature and the second shape feature; and a model training module configured to train the image migration model based on a difference between the second migration image and the third image.

[0009] Another aspect of the present disclosure provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the image migration method and / or the training method of the image migration model provided by the present disclosure.

[0010] According to another aspect of the present disclosure, a non-transitory computer readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to perform the image migration method and / or the training method of the image migration model provided by the present disclosure.

[0011] According to another aspect of the present disclosure, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the image migration method and / or the training method of the image migration model provided by the present disclosure.

[0012] It should be understood that the matters described in this section are not intended to identify key or important features of the embodiments of the present disclosure, nor are they used to limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0013] The accompanying drawings are used to better understand the present scheme and do not limit the present disclosure. Among them:

[0014] Figure 1 is an application scenario diagram of the image migration method and the training method of the image migration model according to the embodiments of the present disclosure.

[0015] Figure 2 is a flowchart of the image migration method according to the embodiments of the present disclosure.

[0016] Figure 3 is a principle diagram of extracting the first attribute feature according to the embodiments of the present disclosure.

[0017] Figure 4 is a principle diagram of extracting the first shape feature according to the embodiments of the present disclosure.

[0018] Figure 5 is a principle diagram of obtaining the first image according to the embodiments of the present disclosure.

[0019] Figure 6 is a principle diagram of the image migration method according to the embodiments of the present disclosure.

[0020] Figure 7 is a flowchart of the training method of the image migration model according to the embodiments of the present disclosure.

[0021] Figure 8 is a principle diagram of the training method of the image migration model according to the embodiments of the present disclosure.

[0022] Figure 9 is a structural block diagram of the image migration device according to the embodiments of the present disclosure.

[0023] Figure 10 is a structural block diagram of the training device of the image migration model according to the embodiments of the present disclosure; and

[0024] Figure 11 is a structural block diagram of an electronic device for implementing the image migration method and / or the training method of the image migration model according to the embodiments of the present disclosure. DETAILED DESCRIPTION

[0025] Exemplary embodiments of the present disclosure are described herein with reference to the accompanying drawings, which are meant to be exemplary in nature, and include various details intended to facilitate understanding of the present disclosure. However, those skilled in the art will recognize that the exemplary embodiments described herein can be practiced with variation of the details as not to depart from the scope and spirit of the present disclosure. Similarly, it will be appreciated that the description set forth herein is not intended to limit the present disclosure to particular devices or methods, since the scope of the present disclosure is commensurate with the appended claims.

[0026] The present disclosure provides an image migration method, which includes an attribute extraction stage, a shape extraction stage, an identity extraction stage, and an image generation stage. In the attribute extraction stage, a first attribute feature of a first object included in a first image is extracted. In the shape extraction stage, a first shape feature of a first target part included in the first object is extracted according to audio data of the first image. In the identity extraction stage, a first identity feature of a second object included in a second image is extracted. In the image generation stage, a first migration image is generated based on the first attribute feature, the first shape feature, and the first identity feature, the first migration image including the second object having the first attribute feature and the first shape feature.

[0027] The following will be described in conjunction with Figure 1 The application scenarios of the method and device provided by the present disclosure are described.

[0028] Figure 1 The application scenario diagram of the image migration method and the training method and device of the image migration model according to the embodiments of the present disclosure is shown.

[0029] As Figure 1 shown, the application scenario 100 of the embodiments can include an electronic device 110, which can be any electronic device with processing functions, including but not limited to a smart phone, a tablet computer, a laptop computer, a desktop computer, a server, and the like.

[0030] The electronic device 110 can perform expression migration on the input image 120 of the first face and the image 130 of the second face, for example, to generate an image 140. The face in the image 140 is the first face, but the first face has the expression and pose of the second face. For example, an identity feature can be extracted from the image 120 of the first face, an attribute feature can be extracted from the image 130 of the second face, the identity feature and the attribute feature can be fused, and a generation network can be used to generate the image 140. The generation network can be a generator in a generative adversarial network (GAN), for example. The attribute feature can include expression features and pose features, for example.

[0031] In an embodiment, the image migration model can be employed to migrate the expression and pose of the second face to the first face, so as to generate the image 140.

[0032] According to an embodiment of the present disclosure, as Figure 1 indicated, the application scenario 100 can further include a server 150. The electronic device 110 can be in communication connection with the server 150 through a network, which can include a wireless or wired communication link.

[0033] Exemplarily, the server 150 can be configured to train the image migration model, and in response to a model acquisition request sent by the electronic device 110, send the trained image migration model 160 to the electronic device 110, so as to facilitate the electronic device 110 to migrate the expression and pose of the face. In an embodiment, the electronic device 110 can further send the image 120 of the first face and the image 130 of the second face to the server 150 through the network, and the server 150 can generate the image 140 according to the trained image migration model.

[0034] According to an embodiment of the present disclosure, as Figure 1 indicated, the application scenario 100 can further include a database 170, which can maintain a large number of images or videos. The server 150 can access the database 170, and extract part of the images with the same face from the database 170 as sample images, and train the image migration model.

[0035] It should be noted that the image migration method provided by the present disclosure can be executed by the electronic device 110 or the server 150. Correspondingly, the image migration apparatus provided by the present disclosure can be arranged in the electronic device 110 or the server 150. The training method of the image migration model provided by the present disclosure can be executed by the server 150. Correspondingly, the training apparatus of the image migration model provided by the present disclosure can be arranged in the server 150.

[0036] It should be understood that Figure 1 the number and type of the electronic devices, servers and databases in the above-mentioned embodiments are only illustrative. According to the implementation needs, there can be any number and type of terminal devices, servers and databases.

[0037] The image migration method provided by the present disclosure will be described in detail below in combination with Figure 1 and Figures 2-6 .

[0038] Figure 2 is a flowchart of the image migration method according to an embodiment of the present disclosure.

[0039] As Figure 2As shown, the image migration method 200 of this embodiment can include operation S210 to operation S240.

[0040] In operation S210, a first attribute feature of a first object included in a first image is extracted.

[0041] According to an embodiment of the present disclosure, the first object may, for example, be a human face, or may be a building, etc. The first image may, for example, be an image obtained by photographing the first object, or may be an image obtained by processing a photographed image, or may be a video frame including the first object in a video. The first image may, for example, be obtained by real-time photographing, or may be obtained by pre-photographing.

[0042] According to an embodiment of the present disclosure, the first attribute feature may, for example, be used to represent a feature such as a pose, an expression, a gaze, a skin state, etc. of the first object.

[0043] In an embodiment, a first encoding network may, for example, be used to extract the first attribute feature from the first image. The first encoding network may, for example, be pre-trained. The first encoding network may, for example, be a lightweight network such as a MobileNet series network, a ShuffleNet network, or a SqueezeNet, etc. The MobileNet network may, for example, be a lightweight deep neural network constructed using a depthwise separable convolution. The first encoding network may, for example, include a convolution layer and a pooling layer.

[0044] In an embodiment, an expression recognition algorithm that fuses a local binary pattern and a local sparse representation may, for example, be used to extract the first attribute feature.

[0045] In an embodiment, the embodiment may, for example, extract an expression feature and a pose feature of the first object respectively, and concatenate the expression feature and the pose feature as the first attribute feature. Alternatively, the first image may, for example, be input into the first encoding network, and the first encoding network may, for example, extract the expression & pose features simultaneously, thereby outputting the first attribute feature.

[0046] In operation S220, a first shape feature of a first target part included in the first object is extracted according to audio data corresponding to the first image.

[0047] According to an embodiment of the present disclosure, the first image may, for example, be a video frame in video data, and the video data may, for example, include audio data corresponding to the first image. The embodiment may, for example, convert the audio data, thereby obtaining an audio feature of the audio data. The audio feature may, for example, be a mel-frequency cepstral coefficient, etc., and the present disclosure does not limit the audio feature. After obtaining the audio feature, the first shape feature may, for example, be extracted from the audio feature. The first shape feature may, for example, be extracted from the audio feature using a temporal convolution network, etc.

[0048] For example, in a case where the first object is a human face, the first target part can be, for example, a mouth. By separately extracting shape features of the mouth, the state of the first object can be better expressed, which is conducive to generating a more realistic transfer image.

[0049] In operation S230, a first identity feature of a second object included in the second image is extracted.

[0050] According to an embodiment of the present disclosure, the second object included in the second image can be the same type of object as the first object described above. For example, the first object and the second object can both be human faces, or the first object and the second object can both be buildings, etc. The first object and the second object can also be the same object, or can be different objects, which are not limited by the present disclosure.

[0051] According to an embodiment of the present disclosure, the first identity feature can be extracted from the second image by using a third encoding network. The third encoding network can use, for example, a residual network (ResNet), a feature pyramid network (FPN), or a U-Net, etc. The third encoding network can include convolutional layers and pooling layers, and has a skip-connection structure, for extracting identity features that are more complex than attribute features.

[0052] In operation S240, a first transfer image is generated based on the first attribute feature, the first shape feature, and the first identity feature, and the first transfer image includes the second object having the first attribute feature and the first shape feature.

[0053] According to an embodiment of the present disclosure, the first attribute feature, the first shape feature, and the first identity feature can be concatenated and input into a generation network Generator, and the first transfer image is generated by the generation network Generator based on the concatenated features. The generated first transfer image includes the second object having the first attribute feature and the first shape feature.

[0054] Embodiments of the present disclosure extract object attribute features from the first image, extract shape features of a target part from audio data corresponding to the first image, extract an identity feature of an object from the second image, and generate a transfer image based on the identity feature, the attribute feature, and the shape feature. Compared with related technologies, the embodiments do not need to perform three-dimensional reconstruction of the object, and thus can reduce the computational complexity and complexity of image transfer to a certain extent. Furthermore, by separately extracting shape features of the target part, the features representing the state of the object can be more abundant, which is conducive to improving the accuracy and authenticity of the obtained transfer image, and thus can improve the user experience.

[0055] According to an embodiment of the present disclosure, when the first migration image is generated based on the first attribute feature, the first shape feature and the first identity feature, the first attribute feature, the first shape feature and the first identity feature can be fused in other ways than splicing, so as to obtain a fusion feature. Then the fusion feature is input into the generation network, and the first migration image is output by the generation network. For example, the first attribute feature, the first shape feature and the first identity feature can be fused by an add operation, or the first attribute feature, the first shape feature and the first identity feature can be spliced first and then convolved by a convolution layer, so as to obtain the fusion feature. It can be understood that the above method of fusing the first attribute feature, the first shape feature and the first identity feature is only an example for understanding the present disclosure, and the present disclosure is not limited thereto. Through the fusion of the features, the generation network can generate a more realistic migration image.

[0056] Figure 3 FIG. 1 is a schematic diagram of a principle of extracting a first attribute feature according to an embodiment of the present disclosure.

[0057] In an embodiment, the extracted first attribute feature can include a first expression feature. For example, the expression feature can be extracted by directly using the expression recognition algorithm or the first encoding network described above.

[0058] According to an embodiment of the present disclosure, the expression feature can be extracted according to the image of the region other than the region where the first target part is located. This is because the target part generally affects the correct recognition of the expression to some extent, and through the method of this embodiment, the extraction accuracy of the expression feature can be improved.

[0059] In an embodiment, the extracted first attribute feature can include a first posture feature. The posture feature can represent, for example, the inclination angle of the first object, etc. The extraction method of the first posture feature is similar to that of the first expression feature.

[0060] According to an embodiment of the present disclosure, the posture feature can be extracted according to the image of the region other than the region where the first target part is located and the region where the second target part is located. The second target part can be, for example, the part such as the eye that affects the posture recognition. In this way, the influence of the first target part and the second target part on the posture recognition can be avoided, and through the method of this embodiment, the extraction accuracy of the posture feature can be improved.

[0061] In an embodiment, as shown in FIG. 3, the first object is a face, the first target part is a mouth, and the second target part is an eye, which are taken as examples to describe the principle of extracting the first attribute feature. Figure 3

[0062] ​In extracting the first expression feature, the region where the first target part is located in the first image can be determined based on the position information of the first target part included in the first object in the first image 301. For example, the position information of the first target part in the first image 301 can be determined by target detection, and the image 302 of the region where the first target part is located can be obtained. Subsequently, the image of the region other than the region where the first target part is located is taken as the first sub-image 303. For example, the detection box surrounding the first target part can be determined in the region surrounded by the first image 301. Subsequently, the center point of the surrounded region is kept unchanged, and the size is expanded to a predetermined size to obtain an expanded detection box. The region surrounded by the expanded detection box in the first image 301 is taken as the region where the first target part is located.

[0063] For example, the first sub-image 303 can also be obtained by assigning the pixel value of the region where the first target part is located in the first image 301 as 0, and keeping the pixel value of other regions unchanged.

[0064] For example, the first expression feature 304 can be extracted according to the first sub-image 303. For example, the aforementioned first encoding network can include two encoding sub-networks. This embodiment can extract the first expression feature 304 by using the third encoding sub-network 310 in the two encoding sub-networks. The third encoding sub-network can be the ResNet network described above or the like.

[0065] In an embodiment, while determining the region where the first target part is located in the first image 301, the region where the second target part is located in the first image 301 can also be determined based on the position information of the second target part in the first image 301, so as to obtain the image 305 of the region where the second target part is located. The determination method of the region where the second target part is located is similar to the determination method of the region where the first target part is located, which will not be described here. After obtaining the region where the first target part is located and the region where the second target part is located, the image of the region other than the region where the first target part is located and the region where the second target part is located in the first image 301 can be taken as the third sub-image 306.

[0066] For example, the pixel value of the region where the first target part is located and the pixel value of the region where the second target part is located in the first image 301 can also be assigned as 0, and the pixel value of other regions is kept unchanged, so as to obtain the third sub-image 306.

[0067] For example, the first pose feature 307 can be extracted according to the third sub-image 306. For example, the embodiment can employ the fourth encoding sub-network 320 in the two encoding sub-networks included in the first encoding network to extract the first pose feature 307. The fourth encoding sub-network 320 may, for example, be the lightweight network described above. The structure of the fourth encoding sub-network can be similar to that of the third encoding sub-network, except that the sample data used in training is different.

[0068] In an embodiment, when extracting the first expression feature, the image of the region where the first target part is located in the image in the video data that is close in playing time to the first image can be employed to replace the image of the region where the first target part is located in the first image. In this way, the influence of the first target part on the expression feature can be considered to a certain extent when extracting the expression feature, facilitating improvement of the accuracy of the extracted expression feature.

[0069] For example, the first image 301 can include a video frame in the video data. In this example, the target image including the first object in the video data can be determined while the first sub-image 303 is obtained. The target image is any image including the first object and having a difference in playing time from the first image that is less than a predetermined value. Subsequently, the image of the region where the first target part is located in the target image can be determined as the second sub-image. The method of obtaining the second sub-image is similar to the method of obtaining the image 302 of the region where the first target part is located described above, and will not be described again here. After the second sub-image is obtained, the second sub-image and the first sub-image 303 described above can be fused to obtain a fused image. Finally, the first expression feature is extracted according to the fused image.

[0070] In this way, the pixel values of the pixels in the second sub-image can be directly assigned to the pixels in the region of the first sub-image 303 having a pixel value of 0, thereby obtaining the fused image. Before the fusion, the second sub-image can be scaled so that the size of the second sub-image is equal to the size of the region of the first sub-image 303 having a pixel value of 0, and then the pixel values of the second sub-image are assigned to the first sub-image.

[0071] It can be understood that, since the first shape feature is considered when generating the migration image, the fusion of the second sub-image and the first sub-image does not cause a loss in the generation accuracy of the migration image.

[0072] Figure 4 is a schematic diagram of the principle of extracting the first shape feature according to an embodiment of the present disclosure.

[0073] In an embodiment, when extracting the first shape feature, in addition to considering the audio data, the image of the first target part can also be considered, so as to improve the accuracy of the extracted first shape feature.

[0074] According to an embodiment of the present disclosure, in the case of extracting the first shape feature by using the second encoding network, the second encoding network can include a first encoding sub-network and a second encoding sub-network. The first encoding sub-network can be used to extract the shape feature from the image of the first target part, and the second encoding sub-network can be used to extract the shape feature from the audio data. Finally, the two parts of the extracted shape features are fused to obtain the first shape feature.

[0075] As shown in FIG. 4, in this embodiment 400, when extracting the first shape feature, the image 402 of the region where the first target part is located in the first image 401 can be determined based on the first position information of the first target part in the first image 401. Then the image 402 can be used to extract the shape feature of the first target part, and the extracted feature can be taken as the first sub-feature. Figure 4 For example, the image of the region where the first target part is located can be input into the first encoding sub-network, and the first shape feature can be output by the first encoding sub-network.

[0076] For example, the mask for the first target part can be generated based on the image 402 of the region where the first target part is located. Then the mask can be used to process the first image 401, so as to obtain the fourth sub-image 403. In the fourth sub-image 403, the pixel values of the pixels in the regions other than the region where the first target part is located are all 0, and the pixel values of the pixels in the region where the first target part is located are the same as the pixel values of the corresponding region in the first image 401. Then the fourth sub-image 403 can be taken as the input of the first encoding sub-network 410, and the first sub-feature 404 can be extracted by the first encoding sub-network 410.

[0077] For example, the first encoding sub-network for extracting the first sub-feature can use a non-lightweight network such as a residual network (ResNet), a feature pyramid network (FPN) or a U-Net, because the accuracy of the shape feature is usually required to be higher, so as to facilitate the generation of a realistic migration image. The encoding network for extracting the first sub-feature can include convolutional layers and pooling layers, and has a skip-layer connection structure.

[0078] In an embodiment, the first sub-feature can also be extracted based on the fourth sub-image by using a boundary feature method or a geometric parameter method.

[0079] At the same time of extracting the first sub-feature, the shape feature of the first target part can also be extracted according to the audio feature of the audio data of the first image, and the shape feature can be taken as the second sub-feature.

[0080]

[0081] ​In an embodiment, the Mel-scale Frequency Cepstral Coefficients (MFCC) or linear spectrum coefficients of the audio data 405 can be extracted first, and the MFCC or linear spectrum coefficients are taken as the audio feature 406 of the audio data. Then, the audio feature 406 can be input into the second encoding sub-network 420, and the second sub-feature 407 is output by the second encoding sub-network 420. The second encoding sub-network 420 can be a convolutional neural network. Specifically, the second encoding sub-network 420 can be a time convolution network.

[0082] After obtaining the first sub-feature 404 and the second sub-feature 407, the two sub-features can be processed by homogenization, and the first shape feature 408 is obtained. Specifically, the elements in the two sub-features can be added and then divided by 2, and the result is taken as the elements in the first shape feature 408. Alternatively, the two sub-features can be pre-assigned with weight values, and the weighted sum of the two sub-features is taken as the first shape feature 408. Alternatively, the second encoding network can further include a fusion network, and the first shape feature 408 is output by the fusion network by inputting the two sub-features into the fusion network. The fusion network can be a fully connected network, or the fusion network can perform the homogenization processing or weighting operation as described above, which is not limited in the present disclosure.

[0083] Figure 5 FIG. 1 is a schematic diagram of obtaining a first image according to an embodiment of the present disclosure.

[0084] According to an embodiment of the present disclosure, the first image described above can be an image cropped according to the position of the first object, for example, so as to avoid the interference of other objects in the image on the extraction of the first attribute feature, and meanwhile, the computational complexity of extracting the attribute feature can be reduced.

[0085] For example, the image collected in real time or pre-collected can be taken as a template image of the object attribute. After obtaining the template image, the embodiment can first detect the template image including the first object, and obtain the second position information of the first object in the template image. Then, the template image is cropped based on the second position information, and the first image is obtained.

[0086] For example, target detection can be performed on the template image to obtain the position information of the detection box surrounding the first object within the template image. This position information may include, for example, the coordinates of the center point of the detection box in a coordinate system constructed based on the template image, as well as the height and width of the detection box. After obtaining this position information, the template image can be cropped according to a predetermined height and width, using the center point of the detection box as the center point. The predetermined height should be greater than the height of the detection box, and the predetermined width should be greater than the width of the detection box. For example, the size of the cropped image may be 256 pixels × 256 pixels. This embodiment can use the cropped image as the first image.

[0087] In one embodiment, after cropping the template image, the background image of the cropped image can be removed, thereby reducing the interference of the background image on the extraction of the first attribute features and the first shape features, thereby further improving the accuracy of the extracted first attribute features and the first shape features, reducing the computational load of extracting attribute features and shape features, and facilitating the generation of more realistic migration images.

[0088] like Figure 5 As shown, in this embodiment 500, when obtaining the first image, object detection can be performed on the obtained template image 501. For example, the template image 501 can be input into the object detection model 510, and the second position information 502 can be obtained through the object detection model 510. The template image 501 can then be cropped based on the second position information, and the cropped image 503 can be used as an intermediate image. The object detection model can include, for example, any one of various models such as the Faster R-CNN model, the SingleShot Multibox Detector (SSD) model, and the You Only Look Once (YOLO) detection model. This disclosure does not limit the specific model.

[0089] After obtaining the intermediate image 503, this embodiment can further perform segmentation processing on the intermediate image to obtain the third location information of the target object in the intermediate image. The target object includes the aforementioned first object. For example, if the first object is a face, then the target object can be a human body.

[0090] For example, such as Figure 5As shown, the intermediate image 503 can be used as input to the image segmentation model 520, and processed by the image segmentation model 520 to obtain the third position information 504. The third position information 504 should include the position of the boundary line of the target object in the intermediate image 503. The image segmentation model may include, for example, a fully convolutional network (FCN) model, a deep mask model, a Snake model, or a BodyPix model, etc., and this disclosure is not limited thereto.

[0091] Based on the obtained third position information 504, a mask image 505 for the target object can be generated. Specifically, the pixel values ​​of all regions in the intermediate image 503 except the region enclosed by the third position information 504 can be adjusted to 0, while the pixel values ​​within the region enclosed by the third position information 504 can be set to 255, thereby obtaining the mask image 505.

[0092] After obtaining the mask image 505, the background image of the intermediate image 503 can be removed based on the mask image 505 to obtain the first image. Specifically, the intermediate image 503 and the mask image 505 can be multiplied at the pixel level, and the pixel value of each pixel in the image obtained by the dot product can be divided by 255 to obtain the first image 506.

[0093] like Figure 5 As shown, the first image 506 obtained through the above processing only includes the foreground image, i.e., the image of the target object, while the background of the target object is all black, thus achieving background removal. Therefore, the influence of the background on the extraction of attribute features and shape features can be avoided.

[0094] According to embodiments of this disclosure, the aforementioned second image may be, for example, an image cropped according to the position of the second object, thereby avoiding interference from other objects in the image besides the second object in extracting the first identity feature, and reducing the computational load of identity feature extraction.

[0095] For example, images acquired in real-time or pre-captured can be used as source images for object identification. After obtaining the source image, this embodiment can first detect the source image including the second object to obtain the fourth position information of the second object in the source image. Subsequently, the source image is cropped based on the fourth position information to obtain the second image. It is understood that the principle of cropping the source image to obtain the second image can be similar to the principle of cropping the template image to obtain the first image described above. The size of the second image can also be, for example, 256×256, which will not be elaborated here.

[0096] It should be noted that when obtaining the second image, it may not be necessary to remove the background image. This is because the networks that extract identity features are usually quite complex and can learn the identity features relatively accurately. This learning process is less affected by the background image. Therefore, the preprocessing steps for the source image can be reduced.

[0097] According to embodiments of this disclosure, when extracting the first identity feature, for example, identity features extracted from multiple images including the second object can be combined. This avoids the problem of inaccurate extracted identity features due to incomplete or unclear identity feature representation in a single image. Thus, the accuracy of the obtained first identity feature can be improved, making the resulting migration image more realistic.

[0098] For example, multiple video frames including the second object can be extracted from a video, and multiple images including the second image can be obtained based on these multiple video frames. For example, the cropping operation described above can be performed on each of the multiple video frames to obtain multiple images including the second object with a size of 256×256. The number of these multiple images can be, for example, 3, 5, 8, or any integer greater than 1, and can be set according to actual needs. This disclosure does not limit this number.

[0099] This embodiment can extract the identity features of the second object included in each of multiple images, thereby obtaining multiple initial identity features. For example, for each of the m images, an identity feature can be obtained through the third coding network described above, serving as an initial identity feature, thus obtaining m initial identity features. Subsequently, the mean feature of these multiple initial identity features can be calculated, that is, the sum of the m initial identity features is divided by m, thereby obtaining the mean feature. Finally, the mean feature of these multiple initial identity features is determined as the first identity feature of the second object.

[0100] Figure 6 This is a schematic diagram illustrating the principle of an image migration method according to an embodiment of the present disclosure.

[0101] According to embodiments of this disclosure, when extracting the first attribute feature, for example, pixel-level image enhancement processing can be performed on the image including the first object to obtain an enhanced image. The first attribute feature is then extracted based on this enhanced image. In this way, the identity features of the first object in the first image can be weakened or even removed, thus improving the extraction accuracy of the first attribute feature, increasing the accuracy and realism of the obtained migration image, and consequently improving the user experience.

[0102] For example, pixel-level image enhancement processing may include adjusting pixel contrast, brightness, blur level, or applying a mosaic effect to pixels. The mosaic effect can degrade the tonal details of pixels and create a disordered color block effect.

[0103] In an embodiment, when the first expression feature is extracted, the enhanced image can also be subjected to a secondary enhancement processing, which can include scaling processing, translation processing, for example. Through the secondary enhancement processing, the extracted first expression feature can be made not to be affected by the image size and angle, so as to improve the accuracy of the first expression feature. Similarly, in the case where the image of the first target part is considered when the first shape feature is extracted, the embodiment can determine the image of the first target part based on the image subjected to the secondary enhancement processing.

[0104] The embodiment can be provided with a data enhancement tool, and after the first image is input into the data enhancement tool, the data enhancement tool can randomly adopt one or more of the aforementioned multiple data enhancement processing manners to implement data enhancement processing on the first image and output an enhanced image.

[0105] According to an embodiment of the present disclosure, when the first attribute feature is extracted, the first image can be subjected to normalization processing first, so as to reduce the calculation amount when the first attribute feature is extracted and improve the image migration efficiency. After the normalization processing is performed, a first normalized image can be obtained, and then the first attribute feature can be extracted based on the first normalized image.

[0106] For example, the normalization processing performed on the first image can be that the pixel value of each pixel in the first image is divided by 255, so that the pixel value of each pixel obtained belongs to the value interval [0, 1], thereby obtaining the first normalized image.

[0107] Similarly, when the first identity feature is extracted, the second image can also be subjected to normalization processing, so as to reduce the calculation amount when the identity feature is extracted and improve the image migration efficiency. After the normalization processing is performed, a second normalized image can be obtained, and then an encoding network can be used to encode the second normalized image to obtain the first identity feature. Specifically, the second normalized image can be input into a third encoding network, and after being processed by the third encoding network, the first identity feature is output. The method of normalizing the second image can be similar to the method of normalizing the enhanced image described above, and will not be described here.

[0108] In an embodiment, when the first attribute feature is extracted, the first image can also be subjected to data enhancement processing first, and then the enhanced image can be subjected to normalization processing.

[0109] For example, as shown in FIG. 4, the first image can be subjected to data enhancement processing first, and then the enhanced image can be subjected to normalization processing. Figure 6As shown, in the image migration method of the embodiment 600, the first image can be the image obtained after removing the background of the intermediate image as described above. The first encoding network can include, for example, a pose encoder 611 (the fourth encoding sub-network described above) and an emotion encoder 612 (the third encoding sub-network described above). The second encoding network can include a content encoder 613 (the first encoding sub-network described above) and an audio encoder 614 (the second encoding sub-network described above). The third encoding network can be, for example, an ID encoder 620, which can employ the ResNet network described above. The pose encoder is configured to extract first pose features, and the emotion encoder is configured to extract first emotion features. The content encoder is configured to extract first sub-features, which can reflect the content of the speech to some extent, and the audio encoder is configured to extract second sub-features. o In the embodiment 600, the first image 601 can be subjected to the data enhancement process described above, and then third, first, and fourth sub-images 601-1, 601-2, and 601-3 can be obtained according to the region where the first target part is located and the region where the second target part is located. The third, first, and fourth sub-images 601-1, 601-2, and 601-3 can be normalized images, for example. The three images are input into the pose encoder 611, the emotion encoder 612, and the content encoder 613, respectively, and are processed by the pose encoder 611, the emotion encoder 612, and the content encoder 613, respectively, to output first pose features 604, first emotion features 605, and first sub-features, respectively. Meanwhile, the audio data corresponding to the first image can be converted to obtain MFCCs 602 as audio features. The MFCCs 602 can then be input into the audio encoder 614 to obtain second sub-features. The first shape features 606 can be obtained by fusing the first sub-features and the second sub-features. Meanwhile, the second image 603 can be preprocessed, for example, by normalization, and the preprocessed image can be input into the ID encoder 620 to output first ID features 607.

[0110] In the embodiment 600, the first image 601 can be subjected to the data enhancement process described above, and then third, first, and fourth sub-images 601-1, 601-2, and 601-3 can be obtained according to the region where the first target part is located and the region where the second target part is located. The third, first, and fourth sub-images 601-1, 601-2, and 601-3 can be normalized images, for example. The three images are input into the pose encoder 611, the emotion encoder 612, and the content encoder 613, respectively, and are processed by the pose encoder 611, the emotion encoder 612, and the content encoder 613, respectively, to output first pose features 604, first emotion features 605, and first sub-features, respectively. Meanwhile, the audio data corresponding to the first image can be converted to obtain MFCCs 602 as audio features. The MFCCs 602 can then be input into the audio encoder 614 to obtain second sub-features. The first shape features 606 can be obtained by fusing the first sub-features and the second sub-features. Meanwhile, the second image 603 can be preprocessed, for example, by normalization, and the preprocessed image can be input into the ID encoder 620 to output first ID features 607.

[0111] After obtaining the first identity feature 607, the first pose feature 604, the first expression feature 605 and the first shape feature 606, the four features can be spliced via a concat() function to obtain a spliced feature 608. Then the spliced feature 608 is input into a multilayer perceptron (MLP) 630 to realize fusion of the first identity feature 607, the first pose feature 604, the first expression feature 605 and the first shape feature 606. Then the output of the multilayer perceptron 630 is taken as an input of a generator network Generator 640, and a migration image 609 in which a second object in the second image 603 has an expression, a pose and a shape of a mouth of a first object in the first image 601 is output by the Generator 640.

[0112] In an embodiment, the extracted first pose feature can be a 12-dimensional vector, the extracted first shape feature can be a 512-dimensional vector, the extracted first expression feature can be a 256-dimensional vector, and the extracted first identity feature can be a 512-dimensional vector, and the spliced feature 507 is a 1292-dimensional vector. The output feature of the multilayer perceptron 630 is also a 1292-dimensional vector, for example. The size of the generated migration image 609 can be equal to the size of the first image and the second image, and can be an image with a size of 256x256, for example.

[0113] Based on the image migration method provided in the disclosure, the disclosure further provides a training method of an image migration model. The following will be described in combination with Figures 7-8 The training method will be described in detail.

[0114] Figure 7 is a flowchart of the training method of the image migration model according to an embodiment of the disclosure.

[0115] According to an embodiment of the disclosure, as Figure 7 shown, the training method 700 of the image migration model of the embodiment can include operation S710 to operation S750, wherein the image migration model can include a first encoding network, a second encoding network, a third encoding network and a generation network.

[0116] In operation S710, a second attribute feature of a third object included in a third image is extracted by using the first encoding network. It can be understood that the implementation of operation S710 is similar to the implementation of obtaining the first attribute feature by using the first encoding network as described above, and will not be described here.

[0117] At operation S720, a second shape feature of a first target part included in the third object is extracted according to the audio data for the third image by using the second encoding network. It can be understood that the implementation of operation S720 is similar to the implementation of the previous operation of obtaining the first shape feature by using the second encoding network, and thus will not be described herein again.

[0118] At operation S730, a second identity feature of the third object included in the fourth image is extracted by using the third encoding network. The implementation of operation S730 is similar to the implementation of the previous operation of obtaining the first identity feature, and thus will not be described herein again. It should be noted that if the third image and the fourth image both include a face, the face in the third image and the face in the fourth image are faces of the same person.

[0119] At operation S740, a second transfer image is generated by using the generation network based on the second attribute feature, the second shape feature, and the second identity feature. The second transfer image includes the third object with the second attribute feature and the second shape feature. The implementation of operation S740 is similar to the implementation of the previous operation of obtaining the first transfer image by using the generation network, and thus will not be described herein again.

[0120] At operation S750, the image transfer model is trained based on the difference between the second transfer image and the third image.

[0121] According to an embodiment of the present disclosure, when the third image is an image with the background removed, since the second transfer image has the background, the mask image for the third object obtained based on the third image can be used to remove the background image in the third transfer image first. Then, the image transfer model is trained based on the difference between the second transfer image with the background removed and the third image.

[0122] For example, the difference between images can be obtained by comparing the two images pixel by pixel. For example, the difference can be represented by the L1 loss, the L2 loss, or the smooth L1 loss between the second transfer image and the third image. Then, the image transfer model is trained by using the back propagation algorithm.

[0123] According to embodiments of this disclosure, the second attribute feature may be similar to the first attribute feature, including a second pose feature and a second expression feature. The first encoding network may include a third encoding sub-network and a fourth encoding sub-network to extract expression features and pose features, respectively. The second encoding network may include a first encoding sub-network, a second encoding sub-network, and a fusion sub-network. The operation of extracting the second shape feature described above can first determine the image of the region where the first target part is located in the third image based on the fifth position information of the first target part in the third image. Then, based on the image of the region where the first target part is located, a fifth sub-image is obtained. Subsequently, based on the fifth sub-image, the shape features of the first target part are extracted using the first encoding sub-network to obtain the third sub-feature. Simultaneously, based on the audio features of the audio data for the third image, the shape features of the first target part are extracted using the second encoding sub-network to obtain the fourth sub-feature. Finally, the third and fourth sub-features are fused using a fusion network to obtain the second shape feature.

[0124] Figure 8 This is a schematic diagram illustrating the principle of the training method for an image transfer model according to an embodiment of the present disclosure.

[0125] According to embodiments of this disclosure, when training an image transfer model, in addition to the loss obtained through pixel-by-pixel comparison as described above, the loss of the image generated by the generative network can also be considered. Simultaneously, a discriminative network can be set up corresponding to the generative network. Thus, the generative network and the discriminative network constitute a generative adversarial network, and the generative network is trained by calculating the generative adversarial loss. In this way, the accuracy of the generative network in the trained image transfer model can be improved, making the transferred images generated by the generative network more realistic.

[0126] For example, such as Figure 8 As shown, in the image transfer method of embodiment 800, the third image can be the image obtained after removing the background image of the intermediate image as described above. The first encoding network may include, for example, a pose encoding network (Pose Encoder) 811 (the fourth encoding sub-network described above) and an emotion encoding network (Emotion Encoder) 812 (the third encoding sub-network described above). The second encoding network may include a content encoding network (Content Encoder) 813 (the first encoding sub-network described above) and an audio encoding network (Audio Encoder). oEncoder 814 (the second encoding sub-network described above). The third encoding network can be, for example, an ID Encoder 820, which can employ the ResNet network described above. A pose encoding network is used to extract the second pose features, and an expression encoding network is used to extract the second expression features. A content encoding network is used to extract the third sub-feature, which can reflect the speech content to some extent, and an audio encoding network is used to extract the fourth sub-feature.

[0127] like Figure 8 As shown, in this embodiment 800, when training the image transfer model, data augmentation processing can be performed on the third image 801 in a similar manner to that described above. Subsequently, based on the regions where the first and second target parts are located, a sixth sub-image 801-1, a seventh sub-image 801-2, and a fifth sub-image 801-3 are obtained. The sixth sub-image 801.1, the seventh sub-image 801.2, and the fifth sub-image 801-3 are similar to the third, first, and fourth sub-images described above, respectively. These sixth sub-images 801-1, the seventh sub-image 801-2, and the fifth sub-image 801-3 can, for example, be normalized images. These three images are then input into the pose encoding network 811, the expression encoding network 812, and the content encoding network 813, respectively. After processing by these networks, a second pose feature 804, a second expression feature 805, and a third sub-feature are output, respectively. Simultaneously, the audio data for the third image 801 can be converted to obtain MFCC 802 as an audio feature. This MFCC 802 can then be input into the audio coding network 814 to obtain a fourth sub-feature. Fusing the third and fourth sub-features yields the second shape feature 806. Simultaneously, the fourth image 803 can undergo preprocessing such as normalization. The preprocessed image is then input into the identity coding network 820, which processes it and outputs the second identity feature 807.

[0128] Subsequently, the spliced ​​feature 808 obtained by concatenating the second pose feature 804, the second shape feature 806, the second expression feature 805, and the second identity feature 807 can be input into the multilayer perceptron 830. The output of the multilayer perceptron 830 is used as the input of the generator network 840, and the generator network 840 outputs the second transfer image 809. It should be noted that when the third image is an image with the background removed, the background can also be removed from the image output by the generator network 840 to obtain the second transfer image 809.

[0129] Subsequently, a first loss 810-1 of the image migration model can be determined based on a pixel-level difference between the second migration image 809 and the third image 801. The first loss can employ the L1 loss described above, etc. Meanwhile, the second migration image 809 and the third image 801 can be input into the discriminator network 850 respectively, and the discriminator network 850 can output a first ground truth probability that the second migration image is a real image and a second ground truth probability that the third image is a real image. The embodiment can determine a generative adversarial loss 810-2 according to the second migration image, the third image, the first ground truth probability, and the second ground truth probability.

[0130] After obtaining the first loss 810-1 and the generative adversarial loss 810-2, the image migration model can be trained according to, for example, the loss of the generative network part in the first loss 810-1 and the generative adversarial loss 810-2. Subsequently, the discriminator can be trained according to the loss of the discriminator part in the generative adversarial loss 810-2, thereby completing one iteration of training.

[0131] In an embodiment, as shown in FIG. 8, in addition to the first loss 810-1 and the generative adversarial loss 810-2 described above, a second loss 810-3 can also be generated according to a comparison between the third image 801 and the second migration image 809 at a feature level. Figure 8

[0132] For example, the third image 801 and the second migration image 809 can be input into a feature extraction network 860, which can be, for example, a VGG network, a ResNet network, etc., and the present disclosure does not limit the same. After extracting the features of the two images, the second loss 810-3 can be calculated according to the difference between the two features. In this embodiment, the second loss 810-3 can be calculated, for example, by using a perceptual loss function, or can also employ the L1 loss, the L2 loss, etc. described above.

[0133] After obtaining the first loss, the generative adversarial loss, and the second loss, a weighted sum of the loss of the generative network part in the first loss and the generative adversarial loss and the second loss can be calculated. The image migration model can be trained according to the weighted sum. Subsequently, the discriminator can be trained based on the loss of the discriminator part in the generative adversarial loss, thereby completing one iteration of training.

[0134] Through multiple iterations of training, the loss of the image migration model can reach a loss threshold, thereby completing the training of the image migration model. The embodiment can further improve the accuracy of the trained image migration model by simultaneously considering the feature difference between the second migration image and the third image when training the image migration model. Therefore, the trained image migration model can generate more realistic migration images.

[0135] ​It can be understood that the second shape feature can be determined by using a method similar to the method described above for determining the first shape feature according to the audio data. The second shape feature can also be determined by using a method similar to the method described above for determining the first shape feature in combination with the audio data and the image data.

[0136] In an embodiment, the first encoding part and the second encoding part described above can also be pre-trained before training the image migration model. The sample data used for pre-training can include image samples and audio samples. Specifically, the image samples can be used as input of the first encoding part to obtain third sub-features. The audio samples can be used as input of the second encoding part to obtain fourth sub-features. Finally, the first encoding part and the second encoding part are trained with the goal of making the third sub-features consistent with the fourth sub-features. For example, the first encoding part and the second encoding part can be trained according to the distance between the third sub-features and the fourth sub-features.

[0137] In an embodiment, the sample data used for pre-training can include both positive samples and negative samples. The image samples in the positive samples correspond to the audio samples. The image samples in the negative samples do not correspond to the audio samples. The image samples in the sample data can be, for example, a continuous image sequence, and the audio samples in the sample data can be, for example, an audio segment corresponding to the length of the image sequence.

[0138] For example, a batch of sample data can include one positive sample and multiple negative samples. For example, this embodiment can use a predetermined loss function to determine the loss, and pre-train the first encoding part and the second encoding part by minimizing the value of the loss.

[0139] The predetermined loss function can be positively correlated with the distance between the two features obtained according to the positive sample, and negatively correlated with the distance between the two features obtained according to the negative sample. For example, a cross-entropy loss function can be used as the predetermined loss function.

[0140] In an embodiment, the predetermined loss function can use the following formula:

[0141]

[0142] wherein, represents the fourth sub-feature; represents the third sub-feature obtained by using the image sample in the positive sample as input of the first encoding part; represents the third sub-feature obtained by taking the image sample in the negative sample as the input of the first encoding part. M can be the number of negative samples in a batch of training data. In an embodiment, a batch of training data can include one positive sample and M negative samples. D() represents the distance between two features.

[0143] In an embodiment, the third sub-feature and the fourth sub-feature obtained can be normalized respectively, and then the loss can be determined according to the distance between the normalized third sub-feature and the normalized fourth sub-feature. For example, the L2 norm normalization method or any normalization method can be used to normalize the third sub-feature and the fourth sub-feature, and the disclosure does not limit the normalization method.

[0144] According to the embodiments of the disclosure, after the first encoding part and the second encoding part are pre-trained, the weight parameters in the second encoding part can not be adjusted when the overall image migration model is trained.

[0145] Based on the image migration method provided by the disclosure, the disclosure further provides an image migration device. The following will be described in detail Figure 9 with reference to the image migration device.

[0146] Figure 9 is a structural block diagram of the image migration device according to the embodiments of the disclosure.

[0147] As Figure 9 shown, the image migration device 900 of the embodiment can include a first attribute extraction module 910, a first shape extraction module 920, a first identity extraction module 930, and a first image generation module 940.

[0148] The first attribute extraction module 910 is configured to extract a first attribute feature of a first object included in the first image. In an embodiment, the first attribute extraction module 910 can be configured to perform the operation S210 described above, and details are not repeated here.

[0149] The first shape extraction module 920 is configured to extract a first shape feature of a first target part included in the first object according to the audio data of the first image. In an embodiment, the first shape extraction module 920 can be configured to perform the operation S220 described above, and details are not repeated here.

[0150] The first identity extraction module 930 is configured to extract a first identity feature of a second object included in the second image. In an embodiment, the first identity extraction module 930 can be configured to perform the operation S230 described above, and details are not repeated here.

[0151] The first image generation module 940 is configured to generate a first migration image based on the first attribute feature, the first shape feature and the first identity feature, the first migration image comprising the second object with the first attribute feature and the first shape feature. In an embodiment, the first image generation module 940 can be configured to perform the operation S230 described above, and details are not repeated here.

[0152] According to an embodiment of the present disclosure, the first attribute feature comprises a first expression feature. The first attribute extraction module 910 comprises: a first image determination sub-module configured to determine, based on the first position information of the first target part in the first image, an image of a region other than the region where the first target part is located in the first image as a first sub-image; and an expression extraction sub-module configured to extract the first expression feature according to the first sub-image.

[0153] According to an embodiment of the present disclosure, the first image comprises a video frame in video data, and the expression extraction sub-module comprises: a first image determination unit configured to determine a target image comprising the first object in the video data, a difference between a play time of the target image and a play time of the first image being less than a predetermined value; a second image determination unit configured to determine an image of a region where the first target part is located in the target image as a second sub-image; an image fusion unit configured to fuse the second sub-image and the first sub-image to obtain a fusion image; and an expression extraction unit configured to extract the first expression feature according to the fusion image.

[0154] According to an embodiment of the present disclosure, the first attribute feature comprises a first posture feature, and the first attribute extraction module 910 comprises: a second image determination sub-module configured to determine, based on the position information of the first target part and the second target part comprised by the first object in the first image, an image of a region other than the region where the first target part is located and the region where the second target part is located in the first image as a third sub-image; and a posture extraction sub-module configured to extract the first posture feature according to the third sub-image.

[0155] According to an embodiment of the present disclosure, the first shape extraction module 920 comprises: a third image determination sub-module configured to determine, based on the first position information of the first target part in the first image, an image of the region where the first target part is located in the first image as a fourth sub-image; a first feature extraction sub-module configured to extract a shape feature of the first target part as a first sub-feature according to the fourth sub-image; a second feature extraction sub-module configured to extract a shape feature of the first target part as a second sub-feature according to an audio feature of the audio data; and a first feature fusion sub-module configured to fuse the first sub-feature and the second sub-feature to obtain the first shape feature, according to the audio data.

[0156] According to an embodiment of the present disclosure, the image migration device 900 can further include a first detection module configured to detect a template image including a first object to obtain second position information of the first object in the template image; and a first cropping module configured to crop the template image based on the second position information to obtain a first image.

[0157] According to an embodiment of the present disclosure, the first cropping module includes a cropping unit configured to crop the template image based on the second position information to obtain a first intermediate image; a segmentation unit configured to perform image segmentation processing on the first intermediate image to obtain third position information of a target object in the first intermediate image, the target object including the first object; a mask generation unit configured to generate a mask image for the target object based on the third position information; and a background removal unit configured to remove a background image in the first intermediate image based on the mask image to obtain the first image.

[0158] According to an embodiment of the present disclosure, the image migration device 900 can further include an image enhancement module configured to perform pixel-level image enhancement processing on an image including the first object to obtain an enhanced image, and the second attribute extraction module is configured to extract the first attribute feature based on the enhanced image.

[0159] According to an embodiment of the present disclosure, the image migration device 900 can further include a second detection module configured to detect a source image including a second object to obtain fourth position information of the second object in the source image; and a second cropping module configured to crop the source image based on the fourth position information to obtain a second image.

[0160] According to an embodiment of the present disclosure, the second image includes a plurality of images, and the first identity extraction module 930 includes an identity extraction submodule configured to extract an identity feature of the second object included in each of the plurality of images to obtain a plurality of initial identity features, and a feature determination submodule configured to determine a mean feature of the plurality of initial identity features as the first identity feature of the second object.

[0161] According to an embodiment of the present disclosure, the first image generation module 940 includes a second feature fusion submodule configured to fuse the first attribute feature, the first shape feature, and the first identity feature to obtain a fusion feature, and an image generation submodule configured to take the fusion feature as an input of a generation network to obtain a first migrated image.

[0162] Based on the training method of the image migration model provided by the present disclosure, the present disclosure further provides a training device of an image migration model. The following will be described in detail in combination with Figure 10 The device will be described in detail.

[0163] Figure 10 is a structural block diagram of the training device of the image migration model according to an embodiment of the present disclosure

[0164] As shown in Figure 10 The training apparatus 1000 of the image migration model of this embodiment can include a second attribute extraction module 1010, a second shape extraction module 1020, a second identity extraction module 1030, a second image generation module 1040, and a model training module 1050. Among them, the image migration model includes a first encoding network, a second encoding network, a third encoding network, and a generation network.

[0165] The second attribute extraction module 1010 is configured to extract second attribute features of a third object included in a third image by using the first encoding network. In an embodiment, the second attribute extraction module 1010 is configured to perform the operation S710 described above, and details are not repeated here.

[0166] The second shape extraction module 1020 is configured to extract second shape features of a first target part included in the third object by using the second encoding network according to audio data of the third image. In an embodiment, the second shape extraction module 1020 is configured to perform the operation S720 described above, and details are not repeated here.

[0167] The second identity extraction module 1030 is configured to extract second identity features of the third object included in a fourth image by using the third encoding network. In an embodiment, the second identity extraction module 1030 is configured to perform the operation S730 described above, and details are not repeated here.

[0168] The second image generation module 1040 is configured to generate a second migration image by using the generation network based on the second attribute features, the second shape features, and the second identity features; the second migration image includes the third object with the second attribute features and the second shape features. In an embodiment, the second image generation module 1040 is configured to perform the operation S740 described above, and details are not repeated here.

[0169] The model training module 1050 is configured to train the image migration model based on a difference between the second migration image and the third image. In an embodiment, the model training module 1050 is configured to perform the operation S750 described above, and details are not repeated here.

[0170] According to an embodiment of the present disclosure, the model training module 1050 comprises: a first loss determination sub-module configured to determine a first loss of the image migration model based on a pixel-level difference between the second migration image and the third image; a ground truth probability determination sub-module configured to input the second migration image and the third image into a discriminative network respectively to obtain a first ground truth probability for the second migration image and a second ground truth probability for the third image; a second loss determination sub-module configured to determine a generative adversarial loss of the image migration model based on the second migration image, the third image, the first ground truth probability, and the second ground truth probability; and a first training sub-module configured to train the image migration model based on the first loss and the generative adversarial loss.

[0171] According to an embodiment of the present disclosure, the model training module 1050 further comprises: a third loss determination sub-module configured to determine a second loss of the image migration model based on a difference between two features extracted from the second migration image and the third image via the feature extraction network. The model training sub-module is configured to train the image migration model based on the first loss, the generative adversarial loss, and the second loss.

[0172] According to an embodiment of the present disclosure, the second encoding network comprises a first encoding sub-network, a second encoding sub-network, and a fusion sub-network. The second shape extraction module 1020 comprises: a fourth image determination sub-module configured to determine an image of a region where the first target part is located in the third image as a fifth sub-image based on fifth position information of the first target part in the third image; a third feature extraction sub-module configured to extract a shape feature of the first target part as a third sub-feature from the fifth sub-image by using the first encoding sub-network; a fourth feature extraction sub-module configured to extract a shape feature of the first target part as a fourth sub-feature from the audio feature of the audio data corresponding to the third image by using the second encoding sub-network; and a third feature fusion sub-module configured to fuse the third sub-feature and the fourth sub-feature by using the fusion network to obtain the second shape feature.

[0173] According to an embodiment of the present disclosure, the training device 1000 of the image migration model described above can further comprise a pre-training module configured to train the first encoding sub-network and the second encoding sub-network before extracting the second shape feature of the first target part included in the third object from the audio data corresponding to the third image by using the second encoding network. The pre-training module comprises: a fifth feature extraction sub-module configured to take an image sample in sample data as an input of the first encoding sub-network to obtain a fifth sub-feature; a sixth feature extraction sub-module configured to take an audio feature of an audio sample in the sample data as an input of the second encoding sub-network to obtain a sixth sub-feature; and a second training sub-module configured to train the first encoding sub-network and the second encoding sub-network according to a difference between the fifth sub-feature and the sixth sub-feature.

[0174] According to an embodiment of the present disclosure, the sample data comprises a first sample and a second sample; the first sample comprises a first image sample and a first audio sample corresponding to the first image sample; the second sample comprises a second image sample and an arbitrary audio sample other than audio corresponding to the second image sample. The second training sub-module comprises: a loss determination unit configured to determine a value of a predetermined loss function according to a difference between the fifth sub-feature and the sixth sub-feature; and a training unit configured to train the first encoding sub-network and the second encoding sub-network according to the value of the predetermined loss function, wherein the value of the predetermined loss function is positively correlated with the first difference and negatively correlated with the second difference, wherein the first difference is a difference between the fifth sub-feature obtained by inputting the first image sample and the sixth sub-feature obtained by inputting the first audio sample; and the second difference is a difference between the fifth sub-feature obtained by inputting the second image sample and the sixth sub-feature obtained by inputting the second audio sample.

[0175] It should be noted that, in the technical solutions of the present disclosure, the acquisition, collection, storage, use, processing, transmission, provision and disclosure of user personal information involved in the technical solutions comply with relevant laws and regulations and do not violate public order and good customs.

[0176] According to an embodiment of the present disclosure, the present disclosure further provides an electronic device, a readable storage medium and a computer program product.

[0177] Figure 11 A schematic block diagram of an example electronic device 1100 that can be used to implement the image migration method and / or the training method of the image migration model according to an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the present disclosure described and / or claimed in this document.

[0178] As shown in Figure 11 The electronic device 1100 includes a computing unit 1101 that can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1102 or a computer program loaded from a storage unit 1108 into a random access memory (RAM) 1103. Various programs and data required for the operation of the electronic device 1100 can also be stored in the RAM 1103. The computing unit 1101, the ROM 1102, and the RAM 1103 are connected to each other through a bus 1104. An input / output (I / O) interface 1105 is also connected to the bus 1104.

[0179] A plurality of components in the device 1100 are connected to the I / O interface 1105, including: an input unit 1106, such as a keyboard, a mouse, etc.; an output unit 1107, such as various types of displays, speakers, etc.; a storage unit 1108, such as a magnetic disk, an optical disk, etc.; and a communication unit 1109, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1109 allows the device 1100 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0180] The computing unit 1101 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1101 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 1101 performs various methods and processes described above, such as the image migration method and / or the training method of the image migration model. For example, in some embodiments, the image migration method and / or the training method of the image migration model can be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as the storage unit 1108. In some embodiments, part or all of the computer program can be loaded and / or installed on the device 1100 via the ROM 1102 and / or the communication unit 1109. When the computer program is loaded to the RAM 1103 and executed by the computing unit 1101, one or more steps of the image migration method and / or the training method of the image migration model described above can be performed. Alternatively, in other embodiments, the computing unit 1101 can be configured to perform the image migration method and / or the training method of the image migration model by any other appropriate means, such as by means of firmware.

[0181] The various embodiments of the systems and techniques described above can be implemented in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a load programmable logic device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0182] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces the functions / operations specified in the flowcharts and / or the block diagrams. The program code can be executed entirely on a machine, partially on a machine, partially on a machine as a stand-alone software package, or entirely on a remote machine or server.

[0183] In the context of the present disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disk drives, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0184] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0185] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0186] The computer system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, also known as cloud computing server or cloud host, which is a host product in the cloud computing service system, to solve the defects of large management difficulty and weak business scalability in traditional physical host and VPS (Virtual Private Server, or VPS for short) services. The server can also be a server of a distributed system, or a server combined with a blockchain.

[0187] It should be understood that various forms of flow shown above can be used with orders of steps reordered, added to, or removed. For example, the steps recited in the present disclosure can be performed in parallel, in series, or in a different order, without limitation herein, as long as the desired results of the technical solutions of the present disclosure can be achieved.

[0188] The above detailed description does not limit the scope of the disclosure. Various modifications, combinations, sub-combinations and alternatives can be made to the detailed description. Any modification, equivalent replacement and improvement etc. made within the spirit and principle of the disclosure shall be included in the scope of the disclosure.

Claims

1. A method for image migration, comprising: extracting a first attribute feature of a first object included in a first image; extracting a first shape feature of a first target part included in the first object according to audio data of the first image; extracting a first identity feature of a second object included in a second image; and generating a first migrated image based on the first attribute feature, the first shape feature and the first identity feature, the first migrated image including the second object with the first attribute feature and the first shape feature; the extracting the first shape feature of the first target part included in the first object according to the audio data of the first image comprises: determining an image of a region in the first image where the first target part is located based on first position information of the first target part in the first image, obtaining a fourth sub-image; extracting a shape feature of the first target part as a first sub-feature according to the fourth sub-image; extracting a shape feature of the first target part as a second sub-feature according to an audio feature of the audio data; and fusing the first sub-feature and the second sub-feature to obtain the first shape feature. the first attribute feature comprises a first expression feature; 2. The method of claim 1, wherein, the extracting the first attribute feature of the first object included in the first image comprises: determining an image of a region in the first image other than the region where the first target part is located based on the first position information of the first target part in the first image, obtaining a first sub-image; and extracting the first expression feature according to the first sub-image. the first image comprises a video frame in video data; the extracting the first expression feature according to the first sub-image comprises: determining a target image including the first object in the video data, a difference between a playing time of the target image and a playing time of the first image being less than a predetermined value; 3. The method of claim 2, wherein, determining an image of a region in the target image where the first target part is located as a second sub-image; fusing the second sub-image and the first sub-image to obtain a fused image; and extracting the first expression feature according to the fused image. the first attribute feature comprises a first posture feature; the extracting the first attribute feature of the first object included in the first image comprises:

4. The method of claim 1, wherein, determining an image of a region in the first image other than the region where the first target part is located and the region where the second target part is located based on position information of the first target part and the second target part included in the first object in the first image, obtaining a third sub-image; and extracting the first posture feature according to the third sub-image.

5. The method of claim 1, further comprising: detecting a template image including the first object, obtaining second position information of the first object in the template image; and cropping the template image based on the second position information to obtain the first image. the cropping the template image based on the second position information to obtain the first image comprises: cropping the template image based on the second position information to obtain a first intermediate image; and cropping the first intermediate image based on the first position information of the first target part in the first image to obtain the first image. ​ 6. The method of claim 5, wherein, ​ ​ performing image segmentation processing on the first intermediate image to obtain third position information of a target object in the first intermediate image, wherein the target object comprises the first object; generating a mask image for the target object based on the third position information; and removing a background image in the first intermediate image based on the mask image to obtain the first image.

7. The method of any one of claims 1-4, wherein, extracting a first attribute feature of a first object included in a first image comprises: performing data enhancement processing on the first image to obtain an enhanced image; and extracting the first attribute feature based on the enhanced image.

8. The method of claim 1, further comprising: detecting a source image comprising the second object to obtain fourth position information of the second object in the source image; and cropping the source image based on the fourth position information to obtain the second image.

9. The method of any one of claims 1-6 and 8, wherein, The second image comprises a plurality of images; and the extracting a first identity feature of a second object included in a second image comprises: extracting an identity feature of the second object included in each of the plurality of images to obtain a plurality of initial identity features; and determining a mean feature of the plurality of initial identity features as the first identity feature of the second object.

10. The method of claim 1, wherein, The generating a first transfer image based on the first attribute feature, the first shape feature, and the first identity feature comprises: fusing the first attribute feature, the first shape feature, and the first identity feature to obtain a fused feature; and inputting the fused feature into a generation network to obtain the first transfer image.

11. A method of training an image translation model, wherein, The image transfer model comprises a first encoding network, a second encoding network, a third encoding network, and a generation network; and the method comprises: extracting a second attribute feature of a third object included in a third image using the first encoding network; extracting a second shape feature of a first target part included in the third object using the second encoding network according to audio data for the third image; extracting a second identity feature of a third object included in a fourth image using the third encoding network; generating a second transfer image based on the second attribute feature, the second shape feature, and the second identity feature using the generation network; the second transfer image comprises the third object with the second attribute feature and the second shape feature; and training the image transfer model based on differences between the second transfer image and the third image; wherein the second encoding network comprises a first encoding subnetwork, a second encoding subnetwork, and a fusion subnetwork; and the extracting a second shape feature of a first target part included in the third object using the second encoding network according to audio data for the third image comprises: determining an image of a region in the third image where the first target part is located to obtain a fifth sub-image based on fifth position information of the first target part in the third image; extracting a shape feature of the first target part using the first encoding subnetwork according to the fifth sub-image to obtain a third sub-feature; and extracting a second shape feature of the first target part using the second encoding subnetwork according to the third sub-feature to obtain a second shape feature of the first target part. extracting, according to an audio feature of audio data of the third image, a shape feature of the first target part by using the second encoding sub-network, to obtain a fourth sub-feature; and fusing the third sub-feature and the fourth sub-feature by using the fusion network to obtain the second shape feature.

12. The method of claim 11, wherein, training the image migration model based on a difference between the second migration image and the third image comprises: determining a first loss of the image migration model based on a pixel-level difference between the second migration image and the third image; inputting the second migration image and the third image into a discriminative network respectively to obtain a first ground truth probability for the second migration image and a second ground truth probability for the third image; determining a generative adversarial loss of the image migration model based on the second migration image, the third image, the first ground truth probability and the second ground truth probability; and training the image migration model based on the first loss and the generative adversarial loss.

13. The method of claim 12, wherein, training the image migration model based on a difference between the second migration image and the third image further comprises: determining a second loss of the image migration model based on a difference between two features extracted from the second migration image and the third image by using a feature extraction network; and training the image migration model based on the first loss, the generative adversarial loss and the second loss.

14. The method of claim 11, further comprising, before extracting, according to an audio feature of audio data of the third image, a second shape feature of a first target part included in the third object by using the second encoding network: obtaining a fifth sub-feature by taking an image sample in sample data as an input of the first encoding sub-network; obtaining a sixth sub-feature by taking an audio feature of an audio sample in the sample data as an input of the second encoding sub-network; and training the first encoding sub-network and the second encoding sub-network according to a difference between the fifth sub-feature and the sixth sub-feature.

15. The method of claim 14, wherein, the sample data comprises a first sample and a second sample; the first sample comprises a first image sample and a first audio sample for the first image sample; the second sample comprises a second image sample and an arbitrary audio sample other than an audio for the second image sample; the training the first encoding sub-network and the second encoding sub-network according to the difference between the fifth sub-feature and the sixth sub-feature comprises: determining a value of a predetermined loss function according to the difference between the fifth sub-feature and the sixth sub-feature; and training the first encoding sub-network and the second encoding sub-network according to the value of the predetermined loss function, The value of the predetermined loss function is positively correlated with the first difference and negatively correlated with the second difference, wherein the first difference is the difference between the fifth sub-feature obtained by inputting the first image sample and the sixth sub-feature obtained by inputting the first audio sample; and the second difference is the difference between the fifth sub-feature obtained by inputting the second image sample and the sixth sub-feature obtained by inputting the audio sample in the second sample.

16. An image migration device, comprising: a first attribute extraction module configured to extract a first attribute feature of a first object included in a first image; a first shape extraction module configured to extract a first shape feature of a first target part included in the first object according to audio data of the first image; a first identity extraction module configured to extract a first identity feature of a second object included in a second image; and a first image generation module configured to generate a first migration image based on the first attribute feature, the first shape feature, and the first identity feature, the first migration image including the second object having the first attribute feature and the first shape feature. The first shape extraction module includes: a third image determination submodule configured to determine an image of a region in which the first target part is located in the first image as a fourth sub-image based on first position information of the first target part in the first image; a first feature extraction submodule configured to extract a shape feature of the first target part as a first sub-feature according to the fourth sub-image; a second feature extraction submodule configured to extract a shape feature of the first target part as a second sub-feature according to an audio feature of the audio data; and a first feature fusion submodule configured to fuse the first sub-feature and the second sub-feature to obtain the first shape feature. The first attribute feature includes a first expression feature.

17. The apparatus of claim 16, wherein, The first attribute extraction module includes: a first image determination submodule configured to determine an image of a region other than the region in which the first target part is located in the first image as a first sub-image based on first position information of the first target part in the first image; and an expression extraction submodule configured to extract the first expression feature according to the first sub-image. The first image includes a video frame in video data; and the expression extraction submodule includes: a first image determination unit configured to determine a target image including the first object in the video data, a difference between a playing time of the target image and a playing time of the first image being less than a predetermined value; 18. The apparatus of claim 17, wherein, a second image determination unit configured to determine an image of a region in which the first target part is located in the target image as a second sub-image; an image fusion unit configured to fuse the second sub-image and the first sub-image to obtain a fused image; and an expression extraction unit configured to extract the first expression feature according to the fused image. The first attribute feature includes a first posture feature. The first attribute extraction module includes:

19. The apparatus of claim 16, wherein, ​ ​ a second image determining sub-module, configured to determine, as a third sub-image, an image of a region of the first image other than a region where the first target part is located and a region where the second target part is located, based on position information of the first target part and the second target part included in the first object in the first image; and a pose extracting sub-module, configured to extract the first pose feature according to the third sub-image.

20. The apparatus of claim 16, further comprising: a first detecting module, configured to detect a template image including the first object, to obtain second position information of the first object in the template image; and a first cropping module, configured to crop the template image based on the second position information to obtain the first image.

21. The apparatus of claim 20, wherein, The first cropping module comprises: a cropping unit, configured to crop the template image based on the second position information to obtain a first intermediate image; a segmentation unit, configured to perform image segmentation processing on the first intermediate image to obtain third position information of a target object in the first intermediate image; the target object includes the first object; a mask generating unit, configured to generate a mask image for the target object based on the third position information; and a background removing unit, configured to remove a background image in the first intermediate image based on the mask image to obtain the first image.

22. The apparatus of any one of claims 16-19, further comprising: an image enhancing module, configured to perform pixel-level image enhancement processing on an image including the first object to obtain an enhanced image, wherein the first attribute extracting module is configured to extract the first attribute feature based on the enhanced image.

23. The apparatus of claim 16, further comprising: a second detecting module, configured to detect a source image including the second object to obtain fourth position information of the second object in the source image; and a second cropping module, configured to crop the source image based on the fourth position information to obtain the second image.

24. The apparatus of any of claims 16-21 and 23, wherein, The second image comprises a plurality of images; and the first identity extracting module comprises: an identity extracting sub-module, configured to extract an identity feature of the second object included in each of the plurality of images respectively to obtain a plurality of initial identity features; and a feature determining sub-module, configured to determine a mean feature of the plurality of initial identity features as a first identity feature of the second object.

25. The apparatus of claim 16, wherein, The first image generating module comprises: a second feature fusing sub-module, configured to fuse the first attribute feature, the first shape feature and the first identity feature to obtain a fused feature; and an image generating sub-module, configured to take the fused feature as an input of a generative network to obtain the first transfer image.

26. An apparatus for training an image translation model, wherein, The image transfer model comprises a first encoding network, a second encoding network, a third encoding network and a generative network; and the apparatus comprises: a second attribute extracting module, configured to extract a second attribute feature of a third object included in a third image by using the first encoding network; a second attribute extracting module, configured to extract a second attribute feature of a third object included in a third image by using the first encoding network; a second shape extraction module configured to extract, by using the second encoding network, a second shape feature of a first target part included in the third object according to audio data of the third image; a second identity extraction module configured to extract a second identity feature of a third object included in a fourth image by using the third encoding network; a second image generation module configured to generate a second transfer image by using the generation network based on the second attribute feature, the second shape feature, and the second identity feature; the second transfer image including the third object having the second attribute feature and the second shape feature; and a model training module configured to train the image transfer model based on a difference between the second transfer image and the third image; wherein the second encoding network includes a first encoding subnetwork, a second encoding subnetwork, and a fusion subnetwork; and the second shape extraction module includes: a fourth image determination submodule configured to determine an image of a region where the first target part is located in the third image as a fifth sub-image based on fifth position information of the first target part in the third image; a third feature extraction submodule configured to extract a shape feature of the first target part as a third sub-feature by using the first encoding subnetwork according to the fifth sub-image; a fourth feature extraction submodule configured to extract a shape feature of the first target part as a fourth sub-feature by using the second encoding subnetwork according to an audio feature of the audio data of the third image; and a third feature fusion submodule configured to fuse the third sub-feature and the fourth sub-feature by using the fusion subnetwork to obtain the second shape feature.

27. The apparatus of claim 26, wherein, the model training module includes: a first loss determination submodule configured to determine a first loss of the image transfer model based on a pixel-level difference between the second transfer image and the third image; a ground truth probability determination submodule configured to input the second transfer image and the third image into a discriminative network respectively to obtain a first ground truth probability for the second transfer image and a second ground truth probability for the third image; a second loss determination submodule configured to determine a generative adversarial loss of the image transfer model based on the second transfer image, the third image, the first ground truth probability, and the second ground truth probability; and a first training submodule configured to train the image transfer model based on the first loss and the generative adversarial loss.

28. The apparatus of claim 27, wherein, the model training module further includes: a third loss determination submodule configured to determine a second loss of the image transfer model based on a difference between two features extracted from the second transfer image and the third image via a feature extraction network, wherein the model training submodule is configured to train the image transfer model based on the first loss, the generative adversarial loss, and the second loss.

29. The apparatus of claim 26, further comprising a pre-training module configured to train the first encoding subnetwork and the second encoding subnetwork before the third object-included first target part of the third image is extracted using the second encoding network according to audio data for the third image; the pre-training module comprising: a fifth feature extraction sub-module configured to obtain fifth sub-features by taking image samples in sample data as input of the first encoding subnetwork; a sixth feature extraction sub-module configured to obtain sixth sub-features by taking audio features of audio samples in the sample data as input of the second encoding subnetwork; and a second training sub-module configured to train the first encoding subnetwork and the second encoding subnetwork according to a difference between the fifth sub-features and the sixth sub-features. The sample data comprises a first sample and a second sample; the first sample comprises a first image sample and a first audio sample for the first image sample; the second sample comprises a second image sample and an arbitrary audio sample other than audio for the second image sample. The second training sub-module comprises: a loss determination unit configured to determine a value of a predetermined loss function according to a difference between the fifth sub-features and the sixth sub-features; and a training unit configured to train the first encoding subnetwork and the second encoding subnetwork according to the value of the predetermined loss function. The value of the predetermined loss function is positively correlated with a first difference and negatively correlated with a second difference, wherein the first difference is a difference between the fifth sub-features obtained by inputting the first image sample and the sixth sub-features obtained by inputting the first audio sample; the second difference is a difference between the fifth sub-features obtained by inputting the second image sample and the sixth sub-features obtained by inputting the audio sample in the second sample.

31. An electronic device comprising: at least one processor; and memory communicatively connected with the at least one processor; wherein the memory stores instructions executable by the at least one processor to cause the at least one processor to perform the method of any one of claims 1-15.

30. The apparatus of claim 29, wherein, The computer instructions are used to cause the computer to perform the method of any one of claims 1-15.

33. A computer program product comprising a computer program which, when executed by a processor, implements the method of any one of claims 1-15. ​ ​ ​ ​ ​ ​ ​ ​ ​ 32. A non-transitory computer readable storage medium having stored thereon computer instructions, wherein, ​ ​

Images