Image processing method, device, equipment and computer readable storage medium

Abstract

Embodiments of the present application provide an image processing method, device and equipment and a computer readable storage medium, wherein the method comprises: obtaining a first image and a second image; performing identity feature extraction on a first target object in the first image to obtain an identity feature vector of the first image; performing posture feature extraction on a second target object in the second image to obtain a posture feature vector of the second image; obtaining an image synthesis vector corresponding to the first image and the second image based on the identity feature vector and the posture feature vector; and generating a synthesis image of the first image and the second image according to the image synthesis vector, wherein the first target object in the synthesis image has the posture of the second target object. Through the embodiments of the present application, the image reproduction adaptability during image synthesis can be improved, and accurate image synthesis processing can be performed on the first image and the second image.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of Internet, and relate to but are not limited to an image processing method, device, equipment and computer readable storage medium. BACKGROUND

[0002] The face reconstruction technology in the related art mainly trains an image processing network by using a big data driven model, sets the identity information of an original image and a target image to be consistent during training of the model, trains a generator network in the image processing network by using a large amount of picture data, and then implements face reconstruction by using the trained image processing network.

[0003] However, the applicant finds in the implementation of the embodiments of the present application that the method in the related art is only limited to reconstruction of a specific target face, the trained image processing network is only applicable to reconstruction of a single target face, is not applicable to other target faces, and requires a large amount of data to support training of a single target face, which makes it impractical to retrain a new image processing network, and since the identity information of an original image and a target image is set to be consistent during most network training, the network has poor adaptability to reconstruction of different identity faces, and has low synthesis accuracy when the network reconstructs faces of different images. SUMMARY

[0004] The embodiments of the present application provide an image processing method, device, equipment and computer readable storage medium, since the identity information of a first image and a second image does not need to be set to be consistent, and an identity feature vector and a posture feature vector are processed to obtain an image synthesis vector, the adaptability of image reconstruction during image synthesis can be improved, and accurate image synthesis processing can be performed on the first image and the second image.

[0005] The technical solution of the embodiments of the present application is as follows:

[0006] The embodiments of the present application provide an image processing method, comprising:

[0007] obtaining a first image and a second image;

[0008] extracting an identity feature of a first target object in the first image to obtain an identity feature vector of the first image;

[0009] extracting a posture feature of a second target object in the second image to obtain a posture feature vector of the second image;

[0010] obtaining an image synthesis vector corresponding to the first image and the second image based on the identity feature vector and the posture feature vector;

[0011] generate a synthesized image of the first image and the second image according to the image synthesis vector; a first target object in the synthesized image has a pose of the second target object.

[0012] The embodiment of the present application provides an image processing device, comprising:

[0013] an obtaining module, configured to obtain a first image and a second image;

[0014] an identity feature extraction module, configured to perform identity feature extraction on a first target object in the first image to obtain an identity feature vector of the first image;

[0015] a pose feature extraction module, configured to perform pose feature extraction on a second target object in the second image to obtain a pose feature vector of the second image;

[0016] a processing module, configured to obtain an image synthesis vector corresponding to the first image and the second image based on the identity feature vector and the pose feature vector;

[0017] a generating module, configured to generate a synthesized image of the first image and the second image according to the image synthesis vector; a first target object in the synthesized image has a pose of the second target object.

[0018] The embodiment of the present application provides an image processing device, comprising:

[0019] a memory, configured to store executable instructions; and a processor, configured to execute the executable instructions stored in the memory to implement the method.

[0020] The embodiment of the present application provides a computer readable storage medium, which stores executable instructions, and is configured to cause a processor to execute the executable instructions to implement the method.

[0021] The embodiment of the present application has the following beneficial effects:

[0022] By performing identity feature extraction on a first target object in a first image and performing pose feature extraction on a second target object in a second image, an identity feature vector and a pose feature vector are obtained correspondingly, and then the identity feature vector and the pose feature vector are processed to obtain a synthesized image of the first image and the second image. In this way, since the identity information of the first image and the second image does not need to be consistent, and the identity feature vector and the pose feature vector are processed, the image reproduction adaptability during image synthesis can be improved, and accurate image synthesis processing can be performed on the first image and the second image. BRIEF DESCRIPTION OF DRAWINGS

[0023] Figure 1is an optional architecture schematic diagram of an image processing system provided by an embodiment of the present application;

[0024] Figure 2 is a structure schematic diagram of a server provided by an embodiment of the present application;

[0025] Figure 3 is an optional flow schematic diagram of an image processing method provided by an embodiment of the present application;

[0026] Figure 4 is an optional flow schematic diagram of an image processing method provided by an embodiment of the present application;

[0027] Figure 5 is an optional flow schematic diagram of an image processing method provided by an embodiment of the present application;

[0028] Figure 6 is an optional flow schematic diagram of an image processing method provided by an embodiment of the present application;

[0029] Figure 7 is an optional flow schematic diagram of an image processing method provided by an embodiment of the present application;

[0030] Figure 8 is an optional flow schematic diagram of a training method of an image processing network provided by an embodiment of the present application;

[0031] Figure 9 is a whole framework flowchart of an image processing method provided by an embodiment of the present application;

[0032] Figure 10 is an implementation process schematic diagram of face landmark preprocessing provided by an embodiment of the present application;

[0033] Figure 11 is a gradient update process schematic diagram of a meta-learning strategy provided by an embodiment of the present application. DETAILED DESCRIPTION

[0034] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings, and the described embodiments should not be regarded as limiting the present application, and all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0035] In the following description, reference is made to "some embodiments", which describe a subset of all possible embodiments, but it is understood that "some embodiments" can be the same subset or different subsets of all possible embodiments, and can be combined with each other as long as there is no conflict. Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application are the same as the meanings understood by those skilled in the art belonging to the technical field of the embodiments of the present application. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application, and are not intended to limit the present application.

[0036] The relevant data collection and processing in the embodiments of the present application should strictly comply with the requirements of relevant laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of authorization of laws and regulations and the personal information subject.

[0037] Before explaining the embodiments of the present application, first explain the professional terms involved in the present application:

[0038] 1) Face reconstruction: Given an original face and a target face, a virtual simulation face is synthesized by a computer, and the identity information of the virtual simulation face is consistent with the identity information of the target face, and the expression information is consistent with the expression information of the original face.

[0039] 2) Meta learning: also known as learning to learn, is to enable computers to have the ability to learn and discover learning rules. When facing new tasks, it can master the rules through a small amount of training.

[0040] 3) Few-shot: refers to providing only a small amount of samples of the target face when performing face reconstruction, for example, generally only a few frames of images are provided for face reconstruction.

[0041] In order to better understand the image processing method provided in the embodiments of the present application, first explain the image processing method in the related art:

[0042] The face reconstruction technology in the related art mainly uses a big data driven method, sets the identity information of the original image and the target image consistent during training, and trains the generator network with a large amount of picture data. For target images that do not appear during training, some technologies can adjust the model in the related art through fine tuning.

[0043] However, the method in the related art is limited to the reproduction of a specific target face, and the trained network is only applicable to the reproduction of a single target face and is not applicable to other target faces. Moreover, the training of a single target face requires a large amount of data support, making it impractical to retrain a new face model. In addition, the reproduction of a new face by fine-tuning the network trained on a large amount of data is feasible, but due to the limited number of new face samples, the effect is general under the given finetune calculation amount. In addition, since the identity information of the original image and the target image is set to be consistent during the training of most networks, the network has poor adaptability to the reproduction of different identity faces, and the synthesis accuracy of the network when reproducing faces of different images is low.

[0044] Based on the above at least one problem existing in the related art, the embodiment of the present application provides an image processing method, first, identity feature extraction is performed on a first target object in a first image to be processed to obtain an identity feature vector of the first image; pose feature extraction is performed on a second target object in a second image to be processed to obtain a pose feature vector of the second image; then, the identity feature vector and the pose feature vector are processed to obtain an image synthesis vector corresponding to the first image and the second image; finally, according to the image synthesis vector, a synthesized image of the first image and the second image is determined. In this way, since the identity information of the first image and the second image does not need to be set to be consistent, and the identity feature vector and the pose feature vector are processed to obtain the image synthesis vector, the image reproduction adaptability during image synthesis can be improved, and accurate image synthesis processing of the first image and the second image can be performed.

[0045] The following describes an exemplary application of the image processing device provided by the embodiment of the present application. The image processing device provided by the embodiment of the present application can be implemented as any terminal with screen display function, such as a notebook computer, a tablet computer, a desktop computer, a mobile device (for example, a mobile phone, a portable music player, a personal digital assistant, a dedicated messaging device, a portable game device), a smart robot, etc., or as a server. In the following, an exemplary application of the image processing device implemented as a server will be described.

[0046] Referring to Figure 1 , Figure 1is an optional architecture schematic diagram of the image processing system 10 provided by the embodiment of the present application. In order to realize the synthesis of the first image and the second image to be processed to obtain a synthesized image with the identity information of the first image and the attitude information of the second image, the image processing system 10 provided by the embodiment of the present application comprises a terminal 100, a network 200 and a server 300. The terminal 100 runs an application program, which can be an image processing application or an image display application. In the implementation of the image processing method of the embodiment of the present application, the terminal 100 displays the first image and the second image to be processed on a current display page 100-1, wherein the first image has a first target object and the second image has a second target object. The terminal 100 sends the first image and the second image displayed on the display page 100-1 to the server 300 through the network 200. The server 300 extracts the identity features of the first target object in the first image to obtain an identity feature vector of the first image, extracts the attitude features of the second target object in the second image to obtain an attitude feature vector of the second image, performs instance normalization processing on the identity feature vector and the attitude feature vector to obtain an image synthesis vector corresponding to the first image and the second image, and determines a synthesized image of the first image and the second image according to the image synthesis vector. After the formation of the synthesized image, the synthesized image is fed back to the terminal 100 through the network 200. After obtaining the synthesized image, the terminal 100 displays the first image, the second image and the synthesized image on the display page 100-1 at the same time for the user to view, or only displays the synthesized image.

[0047] The image processing method related by the embodiment of the present application can also be realized based on artificial intelligence (AI) technology, which is a theory, method, technology and application system for using a digital computer or a machine controlled by a digital computer to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use the knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology of computer science, which attempts to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that the machine has the functions of perception, reasoning and decision-making.

[0048] Artificial intelligence technology is a comprehensive discipline, involving a wide range of fields, both hardware and software technologies. Artificial intelligence basic technologies generally include technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction system, mechatronics, etc. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning, etc.

[0049] Computer vision Computer vision is a science that studies how to make machines "see". More specifically, it refers to using cameras and computers to replace human eyes to identify, track and measure targets, and further process graphics so that the computer processing becomes images more suitable for human observation or transmission to instruments for detection. As a scientific discipline, computer vision researches related theories and technologies, trying to establish artificial intelligence systems that can obtain information from images or multidimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping, and other technologies. It also includes common face recognition, fingerprint recognition and other biometric identification technologies.

[0050] Machine learning Machine learning is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It is a discipline that studies how computers simulate or implement human learning behavior to acquire new knowledge or skills, and reorganize existing knowledge structure to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental approach to making computers intelligent. Its applications are widespread in various fields of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and rule-based learning.

[0051] With the research and progress of artificial intelligence technology, artificial intelligence technology has been researched and applied in many fields, such as common smart home, smart wearable device, virtual assistant, smart speaker, smart marketing, unmanned vehicle, autonomous vehicle, unmanned aerial vehicle, robot, intelligent medical treatment, intelligent customer service, etc. It is believed that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0052] The scheme provided by the embodiments of the present application relates to computer vision technology, machine learning and other technologies of artificial intelligence, which will be described in the following embodiments.

[0053] Figure 2 is a structural schematic diagram of the server 300 provided by the embodiments of the present application, Figure 2 The server 300 shown includes at least one processor 310, a memory 350, at least one network interface 320 and a user interface 330. The various components in the server 300 are coupled together by a bus system 340. It can be understood that the bus system 340 is used to realize the connection and communication between the components. In addition to the data bus, the bus system 340 also includes power buses, control buses and attitude signal buses. However, for the sake of clarity, only the data buses are shown in the figure.Figure 2 Various buses are shown as bus system 340.

[0054] The processor 310 can be an integrated circuit chip that has a processing capability of signals, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, etc., wherein the general-purpose processor can be a microprocessor or any conventional processor.

[0055] The user interface 330 includes one or more output devices 331 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 330 also includes one or more input devices 332 that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0056] The memory 350 can be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard drives, optical drives, etc. The memory 350 optionally includes one or more storage devices remotely located from the processor 310. The memory 350 comprises volatile memory or non-volatile memory, and can also include both volatile and non-volatile memory. Non-volatile memory can be read only memory (ROM), volatile memory can be random access memory (RAM). The memory 350 described in embodiments of the present application is intended to include any suitable type of memory. In some embodiments, the memory 350 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or a subset or superset thereof, which are described below.

[0057] The operating system 351 includes a system program for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0058] The network communication module 352 is used to reach other computing devices via one or more (wired or wireless) network interfaces 320, exemplary network interfaces 320 include Bluetooth, wireless compatibility certification (WiFi), and universal serial bus (USB), etc.

[0059] The input processing module 353 is used to detect and translate one or more user inputs or interactions from one or more input devices 332.

[0060] In some embodiments, the apparatus provided by the embodiments of the present application can be implemented in software, Figure 2 An image processing apparatus 354 stored in the memory 350 is shown, which can be an image processing apparatus in the server 300, which can be software in the form of programs and plug-ins, etc., including the following software modules: an acquisition module 3541, an identity feature extraction module 3542, a posture feature extraction module 3543, a processing module 3544, and a generation module 3545, which are logical, and thus can be combined or further split according to the implemented functions. The functions of each module will be described below.

[0061] In some embodiments, the apparatus provided by the embodiments of the present application can be implemented in software,

[0062] The image processing method provided by the embodiments of the present application will be described below in conjunction with exemplary applications and implementations of the server 300 provided by the embodiments of the present application. Referring to Figure 3 , Figure 3 is an optional flowchart of the image processing method provided by the embodiments of the present application, which will be described in conjunction with the steps shown. Figure 3

[0063] Step S301: acquiring a first image and a second image.

[0064] Here, an image processing request can be received, the image processing request including the first image and the second image, the image processing request being used to request image synthesis on the first image and the second image to obtain a synthesized image having identity information of the first image and posture information of the second image.

[0065] Step S302: performing identity feature extraction on a first target object in the first image to obtain an identity feature vector of the first image.

[0066] ​Here, the first image includes at least one first object, and before image synthesis, a first target object in the first image for providing identity information needs to be determined, and then identity feature extraction is performed on the first target object. In some embodiments, the first target object can be determined according to a focus position of the first image, or the first object closest to the shooting position is determined as the first target object according to the position of each first object in the first image, or the first object with the highest definition is determined as the first target object according to the definition of each first object in the first image, or the user can pre-label the first target object that needs to provide identity information, and the server receives the user's labeling information at the same time of receiving the first image, and determines the first target object according to the labeling information.

[0067] In some other embodiments, if it is detected that there is only one first object in the first image, the one first object can be directly determined as the first target object.

[0068] In some embodiments, the method for determining the first target object can be implemented by using artificial intelligence technology, and through the artificial intelligence technology, the first image is recognized to determine the first object in the first image, and then the first target object is determined from the multiple first objects according to the attribute information (for example, the information such as the focus position, the position, and the definition) of the first object.

[0069] Identity feature extraction refers to extracting the features of a region capable of representing the identity information of the first target object to obtain a feature vector of the region, and synthesizing the features of multiple regions capable of representing the identity information of the first target object, that is, obtaining an identity feature vector of the first image.

[0070] For example, if the first target object is a person, the region capable of representing the identity information can be facial information such as eye, nose, mouth, eyebrow, and face shape, or height, fat, and body shape.

[0071] In step S303, posture feature extraction is performed on the second target object in the second image to obtain a posture feature vector of the second image.

[0072] Here, the second image includes at least one second object, and before image synthesis, a second target object in the second image for providing identity information also needs to be determined. In the embodiments of the present application, the same method as that for determining the first target object can be used to determine the second target object.

[0073] Posture feature extraction refers to extracting the features of a region capable of representing the posture information of the second target object to obtain a feature vector of the region, and determining the obtained feature vector as the posture information of the second target object.

[0074] In the embodiments of the present application, the posture information includes, but is not limited to, facial posture information, gesture information, body posture information, and expression information, etc., wherein the facial posture information includes, but is not limited to, facial expression information. For example, when the identity information of the first image needs to be combined with the expression information of the second image, the extracted posture feature vector of the second image can be a feature vector corresponding to the expression information of the second target object. In the embodiments of the present application, the type of the posture information of the second target object can be pre-set, and in the image synthesis process, the target region of the second target object is determined according to the type of the posture information of the second target object, and then the feature vector of the target region is obtained to obtain the posture feature vector.

[0075] In step S304, the image synthesis vector corresponding to the first image and the second image is obtained based on the identity feature vector and the posture feature vector.

[0076] Here, the identity feature vector and the posture feature vector can be subjected to instance normalization processing to obtain the image synthesis vector of the first image and the second image. Of course, in other embodiments, other processing methods can also be used to synthesize images to obtain the image synthesis vector of the first image and the second image.

[0077] In the embodiments of the present application, the instance normalization processing can be adaptive instance normalization processing. That is, on the basis of the normalization processing, the identity feature vector and the posture feature vector are combined with full connection processing. On the basis of calculating the mean and variance of the identity feature vector and the posture feature vector in the normalization processing, the mean and variance of the identity feature vector and the posture feature vector are obtained through the full connection processing. That is, the mean and variance obtained through the normalization processing are associated with the mean and variance obtained through the full connection processing, so that after the instance normalization processing, the image synthesis vector has the identity information of the first image and also has the posture information of the second image.

[0078] In step S305, a synthesis image of the first image and the second image is generated according to the image synthesis vector.

[0079] Here, the image synthesis vector is converted into a synthesis image according to the information corresponding to the image synthesis vector. The synthesis image is an image that combines the identity information of the first image and the posture information of the second image, that is, the first target object in the synthesis image has the posture of the second target object. For example, the first target object in the first image has an identity A, and the second target object in the second image has an expression B, and the generated synthesis image is an image that has both the identity A and the expression B.

[0080] The image processing method provided in the embodiments of the present application corresponds to obtain the identity feature vector and the posture feature vector by respectively performing identity feature extraction on the first target object in the first image and posture feature extraction on the second target object in the second image, and then performing processing on the identity feature vector and the posture feature vector to obtain the synthetic image of the first image and the second image. In this way, since it is not required that the identity information of the first image and the second image is consistent, and the identity feature vector and the posture feature vector are subjected to instance normalization processing, the image reproduction adaptability during image synthesis can be improved, and accurate image synthesis processing can be performed on the first image and the second image.

[0081] In some embodiments, the image processing system at least includes a terminal and a server, wherein the terminal is configured to display the first image and the second image to be processed, and the server is configured to synthesize the first image and the second image to obtain a synthetic image. Figure 4 An optional flowchart of the image processing method provided in the embodiments of the present application is shown in FIG. 4, and the method includes the following steps: Figure 4

[0082] In step S401, the terminal acquires the first image and the second image.

[0083] Here, the first image and the second image can be images captured by the terminal, images downloaded from a network, or images received from other devices. In the embodiments of the present application, after the first image and the second image are acquired, the first image and the second image are encapsulated in an image processing request, and the image processing request is used to request reproduction of the first image and the second image to synthesize the first image and the second image.

[0084] In some embodiments, the user can also calibrate the first image and the second image through the terminal to calibrate the first target object in the first image that needs to be synthesized and the second target object in the second image that needs to be synthesized.

[0085] In step S402, the terminal sends the image processing request to the server.

[0086] In step S403, the server parses the image processing request to obtain the first image and the second image.

[0087] In step S404, the server performs identity feature extraction on the first target object in the first image to obtain the identity feature vector of the first image.

[0088] In step S405, the server performs posture feature extraction on the second target object in the second image to obtain the posture feature vector of the second image.

[0089] ​It should be noted that the identity feature extraction process in step S404 and the pose feature extraction process in step S405 are similar to the processes in steps S302 and S303 described above, and the embodiments of the present application will not be described again.

[0090] In step S406, the server performs normalization processing on the pose feature vector to obtain a normalized feature vector.

[0091] Here, the normalization processing refers to calculating the mean and variance of all pose feature vectors, and determining the obtained mean and method as the normalized feature vector.

[0092] In step S407, the server performs full connection processing on the normalized feature vector and the identity feature vector to obtain an image synthesis vector corresponding to the first image and the second image.

[0093] After the normalized feature vector is calculated, the normalized feature vector and the identity feature vector are processed by the full connection network, the mean and variance of the normalized feature vector and the identity feature vector are calculated, and the vector corresponding to the mean and variance is determined as the image synthesis vector.

[0094] In step S408, the server generates a synthesized image of the first image and the second image according to the image synthesis vector.

[0095] In step S409, the server sends the synthesized image to the terminal.

[0096] In step S410, the terminal displays the synthesized image on the current interface.

[0097] The image processing method provided by the embodiments of the present application can provide an image processing application, the image processing application runs on the terminal, and the server can be a client server of the image processing application. The user can operate on the client of the image processing application running on the terminal to trigger the image processing process. In the embodiments of the present application, when the user wants to synthesize the first image and the second image, the user operates on the client on the terminal, so that the terminal interacts with the server, and then the image synthesis process is realized, and in the image synthesis process, the identity information of the first image and the second image does not need to be set to be consistent, and the identity feature vector and the pose feature vector are normalized, so that the image reproduction adaptability during image synthesis can be improved, and the first image and the second image can be accurately synthesized, and the user experience is improved.

[0098] Based on Figure 3 , Figure 5 is an optional flowchart of the image processing method provided by the embodiments of the present application, as shown in Figure 5 In some embodiments, step S302 can be implemented by the following steps:

[0099] In step S501, image data corresponding to at least two first marks in the first target object is obtained.

[0100] Here, the first mark refers to any point in the first target object in the first image, and the first mark is used to identify the position of the point. In the first image, the first target object is formed by a large number of point marks connected in series.

[0101] The image data corresponding to the first mark can be the pixel or coordinate of the point corresponding to the first mark.

[0102] In step S502, the image data corresponding to the at least two first marks is normalized to obtain first normalized data.

[0103] Here, the normalization of the image data corresponding to the at least two first marks can be to take the mean and variance of the image data corresponding to the at least two first marks, and the mean and variance are determined as the first normalized data.

[0104] In step S503, the image data mean of the at least two image data is obtained.

[0105] Here, the image data mean is obtained by averaging all the obtained image data.

[0106] In step S504, the identity feature vector of the first image is determined according to the first normalized data and the image data mean.

[0107] Based on Figure 5 , Figure 6 is an optional flowchart of the image processing method provided by the embodiment of the present application, as shown in Figure 6 In some embodiments, step S501 can be implemented by the following steps:

[0108] In step S601, at least two first regions in the first target object are determined.

[0109] Here, the first region can be any region in the first target object, or a region in the first target object that can represent identity information, for example, when the first target object is a face image, the first region can be an eye region, a nose region, and an eyebrow region, etc.

[0110] In step S602, image mark preprocessing is performed on the at least two first regions in the first target object to obtain a first mark for each first region.

[0111] Here, the image mark preprocessing refers to extracting points in the first region to obtain point marks with continuous image data, and determining the point marks as the first marks in the first region. For example, the continuous image data refers to that if the difference between the image data (e.g., pixel value) of an arbitrary point and the image data of an adjacent point is small, it indicates that the image of the region is a continuous image, and thus the image data of the region is valid image data, and the point mark of the region can be determined as the first mark.

[0112] In step S603, image data corresponding to each first mark is obtained.

[0113] Please continue to refer to Figure 6 In some embodiments, step S504 can be implemented by the following steps:

[0114] In step S604, the difference between the first normalized data and the image data mean is determined as the pose information of the first image.

[0115] In step S605, the identity feature vector of the first image is determined according to the image data mean and the pose information of the first image.

[0116] It should be noted that in some embodiments, when determining the pose feature vector of the second image, a method similar to the process of determining the identity feature vector in the embodiments of the present application can also be used. The difference is that after the pose information of the second image is determined, the pose feature vector of the second image is directly obtained according to the pose information, without determining the identity feature vector of the second image according to the image data mean and the pose information of the second image.

[0117] Based on Figure 3 , Figure 7 is an optional flowchart of the image processing method provided by the embodiments of the present application, as Figure 7 shown, in some embodiments, step S303 can be implemented by the following steps:

[0118] In step S701, image data corresponding to at least two second marks in the second target object is determined.

[0119] Here, the determination process of the second mark can be similar to the method in step S602 described above. Image mark preprocessing is performed on at least two second regions in the second target object to obtain second marks of each second region. After the second marks are determined, the corresponding image data is determined according to the positions of the second marks in the second target object.

[0120] In some embodiments, the pose information includes expression information, and correspondingly, step S701 can be implemented by the following steps:

[0121] Step S7011, determine a face image of the second target object.

[0122] Step S7012, in the face image, determine at least two second regions related to the expression information.

[0123] Here, the at least two second regions related to the expression information include but are not limited to: eye regions and eyebrow regions.

[0124] Step S7013, perform image mark preprocessing on the at least two second regions to obtain a second mark of each second region.

[0125] Step S7014, obtain image data corresponding to each second mark.

[0126] Step S702, perform normalization processing on the image data corresponding to the at least two second marks to obtain second normalized data.

[0127] Step S703, obtain an image data mean of the image data corresponding to the at least two second marks.

[0128] Here, obtaining the image data mean is to calculate a mean of the image data corresponding to all of the at least two second marks.

[0129] Step S704, determine a difference between the second normalized data and the image data mean as a pose feature vector of the second image.

[0130] Here, when determining the pose feature vector of the second image, the difference between the second normalized data and the image data mean can be calculated, and the difference is directly determined as the pose feature vector of the second image.

[0131] The image processing method provided by the embodiments of the present application can obtain accurate identity information and pose information through accurate calculation, thereby ensuring more accurate subsequent instance normalization processing, obtaining more accurate image synthesis vectors, and further obtaining more accurate synthesized images, and realizing accurate synthesis of the first image and the second image.

[0132] In some embodiments, the image processing method provided by the embodiments of the present application can also be implemented by using an image processing network trained based on artificial intelligence technology, that is, using the image processing network to determine the synthesized image of the first image and the second image. Alternatively, the first image and the second image can be identified and processed by using artificial intelligence technology to obtain the final synthesized image.

[0133] Figure 8 is an optional flow diagram of a training method of an image processing network provided by the embodiments of the present application, as shown in Figure 8 The training method comprises the following steps:

[0134] In step S801, the first sample image and the second sample image are input into the image processing network, and the first sample label of the first sample image and the second sample label of the second sample image are determined by the label preprocessing model in the image processing network.

[0135] Here, the first sample image and the second sample image are input into the image processing network as sample data, wherein the first sample image is used to provide sample identity information, and the second sample image is used to provide sample pose information.

[0136] In the embodiments of the present application, the first sample label is a point label obtained by image label preprocessing on the first sample image, and the second sample label is a point label obtained by image label preprocessing on the second sample image. In the implementation process, the image label preprocessing model in the image processing network can be used to perform image label preprocessing on the first sample image and the second sample image respectively to obtain the first sample label and the second sample label.

[0137] In step S802, the sample identity feature vector of the first sample image is obtained by performing identity feature extraction on the first sample image by the target identity encoder in the image processing network.

[0138] In some embodiments, step S802 can be implemented by the following steps:

[0139] In step S8021, the first sample label is input into the target identity encoder. In step S8022, the feature vector corresponding to the first sample label is sequentially subjected to at least one downsampling process and at least one convolution process by at least one downsampling layer and at least one convolution layer in the target identity encoder to obtain the sample identity feature vector of the first sample image.

[0140] In step S803, the sample pose feature vector of the second sample image is obtained by performing feature extraction on the second sample label by the feature extraction layer in the image processing network.

[0141] In step S804, the sample synthesis vector corresponding to the first sample image and the second sample image is obtained by performing instance normalization on the sample identity feature vector and the sample pose feature vector by the generator in the image processing network.

[0142] In some embodiments, step S804 can be implemented by the following steps:

[0143] Step S8041, input the sample identity feature vector and the sample pose feature vector into the generator. Step S8042, encode the sample identity feature vector and the sample pose feature vector respectively by the encoder in the generator, and obtain the identity encoding vector and the pose encoding vector correspondingly. Step S8043, perform adaptive instance normalization processing on the identity encoding vector and the pose encoding vector by the adaptive instance normalization layer in the generator, and obtain the sample synthesis vector corresponding to the first sample image and the second sample image.

[0144] Step S805, input the sample synthesis vector into the preset loss model, and obtain the loss result.

[0145] Here, the preset loss model is used to compare the sample synthesis vector with a preset synthesis vector to obtain the loss result, wherein the preset synthesis vector can be a synthesis vector corresponding to the first sample image and the second sample image set by the user in advance.

[0146] In the embodiment of the application, the sample synthesis image can be determined according to the sample synthesis vector, and the preset synthesis image can be determined according to the preset synthesis vector. The preset loss model includes a loss function, and the similarity between the sample synthesis image and the preset synthesis image can be calculated by the loss function. In the calculation process, the similarity between the sample synthesis image and the preset synthesis image can be obtained by calculating the distance between the sample synthesis vector and the preset synthesis vector, and the loss result can be determined according to the similarity. When the distance between the sample synthesis vector and the preset synthesis vector is larger, the similarity between the sample synthesis image and the preset synthesis image is smaller, which indicates that the training result of the model is far from the true value, and further training is needed. When the distance between the sample synthesis vector and the preset synthesis vector is smaller, the similarity between the sample synthesis image and the preset synthesis image is larger, which indicates that the training result of the model is closer to the true value.

[0147] Step S806, according to the loss result, correct the network parameters in the landmark preprocessing model, the target identity encoder and the generator, and obtain the corrected image processing network.

[0148] Here, when the similarity is greater than the preset similarity threshold, the loss result indicates that the landmark preprocessing model in the current image processing network cannot accurately determine the first sample landmark of the first sample image and the second sample landmark of the second sample image, and / or the target identity encoder cannot accurately extract the identity feature of the first sample image, and / or the generator cannot accurately extract the feature of the second sample landmark to obtain the accurate sample pose feature vector of the second sample image. Therefore, the current image processing network needs to be corrected. Then, at least one of the landmark preprocessing model, the target identity encoder and the generator can be corrected according to the similarity until the similarity between the sample synthesis image output by the image processing network and the preset synthesis image meets the preset condition, and the corresponding image processing network is determined as the trained image processing network.

[0149] The training method of the image processing network provided by the embodiments of the present application can correct at least one of the landmark preprocessing model, the target identity encoder and the generator according to the loss result, and the obtained image processing network can accurately synthesize the first image and the second image to obtain a synthesis image meeting the user demand, thereby providing a user experience.

[0150] In some embodiments, step S806 can be implemented by the following steps:

[0151] Step S8061, obtaining a first update gradient of the network parameter at the Nth network update.

[0152] Step S8062, obtaining a second update gradient of the network parameter at the (N+1)th network update. The second update gradient is an update gradient obtained on the basis of the first update gradient.

[0153] Step S8063, determining a target update gradient of the network parameter at the (N+2)th network update as a sum of the first update gradient and the second update gradient.

[0154] Step S8064, correcting the network parameter in the landmark preprocessing model, the target identity encoder and the generator by using the target update gradient to obtain a corrected image processing network.

[0155] In the embodiments of the present application, the gradient of the previous update of the network parameters is taken as the basis for the current update of the network parameters, and the sum of the two consecutive update gradients is determined as the target update gradient for the next parameter update. Through such iterative update, the ordered and directed update of the network parameters can be realized, so that the updated network parameters gradually approach the real parameter value, thereby improving the training accuracy and training efficiency of the network.

[0156] In some embodiments, before performing identity feature extraction on the first sample image by the target identity encoder, the method further includes:

[0157] In step S81, the dimension of the feature vector corresponding to the first sample label is transformed; wherein the dimension of the feature vector after the dimension transformation is the same as the vector dimension of the adaptive instance normalization layer.

[0158] In the following, an exemplary application of the embodiments of the present application in an actual application scenario will be described.

[0159] The embodiments of the present application provide an image processing method, which is a generation method for a face picture. The embodiments of the present application realize a fast and effective face reproduction algorithm. Given an original face picture and a small amount of target face pictures of samples, the embodiments of the present application encode the target face through preprocessing of face landmarks, and reproduce the face picture through a generator network, and the reproduced face picture keeps consistent with the identity information of the target face and consistent with the expression of the original face.

[0160] For example, given an identity picture A and a pose picture B (or an expression picture), the embodiments of the present application can control to generate the face of A with the expression of B. In addition, the embodiments of the present application also support generating a face picture directly specified with a specific expression pose, for example, a user can specify the pose of a face picture by drawing a face landmark by himself / herself, and the embodiments of the present application will generate the face of A with the expression pose specified by the user.

[0161] The embodiment of the present application mainly consists of a face landmark preprocessing network (Landmark Processor), a target face encoder (Target Encoder) and a generator network (Generator). By inputting a small amount of target face picture A to be edited and an original face pose picture B, these pictures are first sent to the face landmark preprocessing network, and each face picture corresponds to a 1*68*2 feature vector, which contains information of eyes, nose, mouth, eyebrows and face shape. The connection result (concatenate) of the target face image and its feature vector is sent to the target face encoding network, which outputs a 1*1*512 encoded identity feature vector. The landmark feature vector of the original face (corresponding to the above-mentioned pose feature vector) is input into the generator network, and the identity feature vector is adjusted through the adaptive instance normalization layer, so that the identity information of the finally generated picture is strengthened. The virtual face picture generated by the embodiment of the present application modifies the expression information of the target face, but largely preserves the original identity information of the target face.

[0162] The embodiment of the present application uses the method of meta-learning to simulate the face reproduction task of limited sample learning purposefully, and to improve the reproduction effect under the limitation of a small amount of samples and calculation amount by making the network learn to adapt to limited sample learning. At the same time, the embodiment of the present application uses the strategy of face landmark conversion to consider the inconsistency of face landmarks during reproduction, slow down the influence of identity gap, and make the generation result more realistic.

[0163] The embodiment of the present application can edit the face picture with the specified facial expression while preserving the identity. The user uploads the identity picture A and the expression picture B to the background server, and the background server uses a specific algorithm to quickly generate a composite image, which has the identity of A and the expression of B, and returns the face image with the identity of A and the expression of B to the user. In addition, the user can also upload the identity picture A and the edited face expression landmark to the background server, and the background server uses a specific algorithm to generate a face image with the identity of A and the expression of B, and returns it to the user.

[0164] The method of the embodiment of the present application includes face landmark preprocessing of input pictures, face generation based on conditional GAN (conditional GAN, Conditional Generative Adversarial Network) and meta-learning strategy. Figure 9 The overall framework flowchart of the image processing method provided by the embodiment of the present application is shown in Figure 9 The target picture (corresponding to the first image) and the original picture (Corresponding to the second image above, due to the original input image) It can support multiple images, therefore, Figure 9 China adopts It means that among them Let i represent the i-th image among K images. Each image is processed by the Landmark Processor 901 to obtain its own face markers. (in, )and The target image is processed using the Target Encoder 902. Face icon Extracting features that represent identity information yields identity information features. Generator 903 uses the original image. Face icon As input, identity information features obtained from the target identity encoder 902 are added through an adaptive instance normalization layer (AdalN) 9031. Adjust the identity information of the generated face to finally generate a face image. (i.e., synthesized images). In generating face images... Then, the generated face images are processed by the discriminator 904. The model is compared with the calibrated ground truth image, thereby correcting the model parameters in the face marker preprocessing network 901, target identity encoder 902 and generator 903 based on the discrimination result.

[0165] In the face icon preprocessing, the embodiments of this application use a face icon preprocessing scheme based on face icon decomposition. Figure 10 This is a schematic diagram illustrating the implementation process of face marker preprocessing provided in the embodiments of this application, as shown below. Figure 10 As shown, given an input image 1001, a 3D face marker 1002 is first obtained through 3D dense face alignment (3DDFA), and then normalized to obtain the normalized face marker. The normalized face markers can be decomposed using the following formula (1-1):

[0166] (1-1);

[0167] in, The mean portion of all face data representing the face marker; The part representing the identity information of the facial recognition feature; represents the expression information part of the face landmark. Thus, the goal of face reenactment is to obtain a face landmark with target identity and original expression wherein the face landmark is represented by the following equation (1-2):

[0168] (1-2);

[0169] wherein, represents the identity information part of the face landmark with target identity ; and represents the expression information part of the face landmark with original expression

[0170] The mean part of all face data can be obtained by taking the average of all data, while the identity information part and the expression information part are predicted by the implementation process of face landmark preprocessing shown in Figure 10

[0171] Please continue to refer to Figure 10 , the input of the multi-layer perceptron (MLP) 1003 is concatenated by the feature vector encoded by the VGGFace network 1004 from the RGB image and the normalized face landmark minus the mean part of the face data . The output of the MLP 1003 is the expression information part ; while the identity information part is obtained from the difference between the mean part of the face data and the expression information part

[0172] In the face generation process based on conditional GAN, the face generation process based on conditional GAN is composed of a target identity encoder E, a generator G and a discriminator D, as shown in the target identity encoder 902, the generator 903 and the discriminator 904. Figure 9

[0173] wherein the target identity encoder: used to receive a target image and its face landmark as input, encode the identity information through a series of down-sampling convolutional layers, and use the feature vector of the last layer of the encoding as the identity feature vector.

[0174] ​​​​​Generator G: Accepts face markers from the original image and identity feature vectors obtained from the target identity encoder E as input. The network structure of generator G is an encoder-decoder. It can encode the face markers from the original image and enhance the identity information of the generated image through an adaptive instance normalization layer during the decoding stage. In some embodiments, before utilizing the identity feature vectors obtained from the target identity encoder E, the feature vector dimensions are first transformed using an MLP to adapt to the dimensionality requirements of the adaptive instance normalization layer.

[0175] Discriminator D: Adopting the idea of ​​conditional GAN, it takes an input image, face markers, and identity ID to determine whether the generated image retains the identity information of the target image and whether it closely resembles the face markers of the original image.

[0176] In this embodiment of the application, the optimization objective function of the entire network is as follows (1-3):

[0177] (1-3);

[0178] in, Represent the objective function; Indicates resistance to loss; Indicates content reconstruction loss; The weight representing the content reconstruction loss; Indicates feature matching loss; The weights represent the feature matching loss; Indicates the encoding matching loss; The weights represent the encoding matching loss.

[0179] The discriminant output of discriminator D This can be expressed by the following formula (1-4):

[0180] (1-4);

[0181] in, This refers to the comparison between the calibrated ground truth image and... Face icon The first discrimination result obtained by performing discrimination; This refers to the generated face image. and Face icon The second discrimination result obtained by performing discrimination; This represents the original input image; This represents the target image that was input. express Facial markers; Indicate the target's identity; These represent the parameters of the entire network. This represents the total set of identity feature vectors.

[0182] The meta-learning strategy of this application embodiment will be described below.

[0183] This application proposes a meta-learning strategy that enables the model to adapt to new characters more quickly. For example... Figure 11 The diagram shown is a schematic of the gradient update process of the meta-learning strategy provided in this application embodiment. Given the pre-trained parameters of the face reconstruction model (the pre-trained parameters can be those in the above formulas (1-4)). This application's embodiments simulate the few-shot learning problem by taking a small number of samples of different new people and updating the parameters of the pre-trained model with a specific amount of update computation. In this application's embodiments, the model parameters can be updated using the Reptile scheme, enabling the updated model to adapt more quickly to the task of reproducing new people given a small number of samples and a specific amount of update computation. Thus, in practical applications, users only need to provide a small number of target people, and the network can achieve better face reproduction within a specific amount of update computation (time).

[0184] like Figure 11 As shown, the pre-trained parameters Pre-trained parameters Indicates weight; Figure 11 The solid lines in the diagram represent the meta-learning process; Figure 11 The dashed lines in the diagram indicate the fine-tuning process.

[0185] like Figure 11 As shown in the left figure, for three different target images 1101, 1102, and 1103, the corresponding pre-trained parameters are... The update gradients are respectively , and , , and The corresponding target update directions are as follows: , and This section uses target image 1101 as an example for explanation, and describes the pre-trained parameters. The goal of training is to improve the network's performance given a small number of target images 1101. During training, the network is updated with a few target images 1101 at a time. The gradient is obtained, i.e., the updated gradient. Then, the gradient is updated. For pre-trained parameters The pre-trained parameters are updated during the update process. Will follow the update direction in the direction of the arrow, so that the pre-trained parameters adapt more quickly to the update direction .

[0186] In the embodiments of the present application, the parameters of the model can be updated through the Reptile scheme, as shown in the right graph of FIG. 11. If the MAML algorithm in the related art is used to calculate the update gradient, the obtained update gradient is Figure 11 ; if the pre-training (Pretrain) algorithm in the related art is used to calculate the update gradient, the obtained update gradient is ; and if the Reptile scheme provided in the embodiments of the present application is used to calculate the update gradient, for the target image 1101 in the left graph, the update gradient . Figure 11

[0187] The embodiments of the present application implement a flexible face reconstruction algorithm, which can quickly generate a synthetic face according to the identity picture and expression picture provided by the user, or can be provided by the user with expression information. The method has potential commercial value.

[0188] It should be noted that for each model and device described in the embodiments of the present application, the specific network structure described in the text is not limited to be used, and other deep neural network structures can also be used to implement the components in the framework. In addition, the meta-learning scheme mentioned in the text is not limited to be used, and other meta-learning schemes are also within the protection scope of the embodiments of the present application.

[0189] The following continues to illustrate an exemplary structure of the image processing apparatus 354 implemented as a software module provided in the embodiments of the present application. In some embodiments, as shown in FIG. 12, the software module stored in the image processing apparatus 354 of the memory 350 can be an image processing apparatus in the server 300, including: Figure 2

[0190] The acquisition module 3541 is configured to acquire a first image and a second image.

[0191] The identity feature extraction module 3542 is configured to perform identity feature extraction on a first target object in the first image to obtain an identity feature vector of the first image.

[0192] The posture feature extraction module 3543 is configured to perform posture feature extraction on a second target object in the second image to obtain a posture feature vector of the second image.

[0193] ​​​The processing module 3544 is configured to obtain an image synthesis vector corresponding to the first image and the second image based on the identity feature vector and the pose feature vector.

[0194] The generating module 3545 is configured to generate a synthesized image of the first image and the second image according to the image synthesis vector, and a first target object in the synthesized image has a pose of the second target object.

[0195] In some embodiments, the identity feature extraction module is further configured to: obtain image data corresponding to at least two first marks in the first target object; perform normalization processing on the image data corresponding to the at least two first marks to obtain first normalized data; obtain an image data mean of the at least two image data; and determine the identity feature vector of the first image according to the first normalized data and the image data mean.

[0196] In some embodiments, the identity feature extraction module is further configured to: determine at least two first regions in the first target object; perform image mark preprocessing on the at least two first regions in the first target object to obtain a first mark of each of the first regions; and obtain the image data corresponding to each of the first marks.

[0197] In some embodiments, the identity feature extraction module is further configured to: determine a difference between the first normalized data and the image data mean as pose information of the first image; and determine the identity feature vector of the first image according to the image data mean and the pose information of the first image.

[0198] In some embodiments, the pose feature extraction module is further configured to: determine image data corresponding to at least two second marks in the second target object; perform normalization processing on the image data corresponding to the at least two second marks to obtain second normalized data; obtain an image data mean of the image data corresponding to the at least two second marks; and determine a difference between the second normalized data and the image data mean as the pose feature vector of the second image.

[0199] In some embodiments, the pose information includes expression information.

[0200] The pose feature extraction module is further configured to: determine a face image of the second target object; determine at least two second regions related to the expression information in the face image; perform image mark preprocessing on the at least two second regions to obtain a second mark of each of the second regions; and obtain the image data corresponding to each of the second marks.

[0201] In some embodiments, the normalization processing module is further configured to: normalize the pose feature vector to obtain a normalized feature vector; and perform full connection processing on the normalized feature vector and the identity feature vector to obtain the image synthesis vector corresponding to the first image and the second image.

[0202] In some embodiments, the apparatus further includes a processing module configured to determine a synthesized image of the first image and the second image using an image processing network.

[0203] The image processing network is trained by the following steps: inputting a first sample image and a second sample image into the image processing network, determining a first sample landmark of the first sample image and a second sample landmark of the second sample image by a landmark preprocessing model in the image processing network; extracting an identity feature of the first sample image by an identity encoder in the image processing network to obtain a sample identity feature vector of the first sample image; extracting a pose feature of the second sample landmark by a feature extraction layer in the image processing network to obtain a sample pose feature vector of the second sample image; performing instance normalization processing on the sample identity feature vector and the sample pose feature vector by a generator in the image processing network to obtain a sample synthesis vector corresponding to the first sample image and the second sample image; inputting the sample synthesis vector into a preset loss model to obtain a loss result; and correcting network parameters in the landmark preprocessing model, the identity encoder, and the generator according to the loss result to obtain a corrected image processing network.

[0204] In some embodiments, the image processing network is trained by the following steps: inputting the first sample landmark into the identity encoder; and sequentially performing at least one down-sampling processing and at least one convolution processing on a feature vector corresponding to the first sample landmark by at least one down-sampling layer and at least one convolution layer in the identity encoder to obtain a sample identity feature vector of the first sample image.

[0205] In some embodiments, the image processing network is trained by the following steps: inputting the sample identity feature vector and the sample pose feature vector into the generator; encoding the sample identity feature vector and the sample pose feature vector by an encoder in the generator to obtain an identity encoding vector and a pose encoding vector, respectively; and performing adaptive instance normalization processing on the identity encoding vector and the pose encoding vector by an adaptive instance normalization layer in the generator to obtain a sample synthesis vector corresponding to the first sample image and the second sample image.

[0206] In some embodiments, the image processing network is trained by the following steps: before identity feature extraction is performed on the first sample image by the target identity encoder, performing dimension transformation processing on the feature vector corresponding to the first sample label; wherein the dimension of the feature vector after dimension transformation processing is the same as the vector dimension of the adaptive instance normalization layer.

[0207] In some embodiments, the image processing network is trained by the following steps: at the Nth network update, obtaining a first update gradient of the network parameters; at the N+1th network update, obtaining a second update gradient of the network parameters; wherein the second update gradient is an update gradient obtained on the basis of the first update gradient; determining the sum of the first update gradient and the second update gradient as a target update gradient of the network parameters at the N+2th network update; and using the target update gradient to correct the network parameters in the landmark preprocessing model, the target identity encoder, and the generator, to obtain the corrected image processing network.

[0208] It should be noted that the description of the device embodiments of the present application is similar to the description of the above method embodiments, has similar beneficial effects as the method embodiments, and therefore will not be described again. For technical details not disclosed in the device embodiments, please refer to the description of the method embodiments of the present application for understanding.

[0209] The computer program product or computer program provided in the embodiments of the present application includes computer instructions stored in a computer readable storage medium. The processor of the computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to make the computer device execute the method described above in the embodiments of the present application.

[0210] The embodiments of the present application provide a storage medium storing executable instructions, wherein the storage medium stores executable instructions. When the executable instructions are executed by a processor, the processor will execute the method provided by the embodiments of the present application, for example, the method shown in the figure. Figure 3

[0211] ​In some embodiments, the storage medium can be a computer-readable storage medium, such as a ferroelectric memory (FRAM), a read only memory (ROM), a programmable read only memory (PROM), an erasable programmable read only memory (EPROM), an electrically erasable programmable read only memory (EEPROM), a flash memory, a magnetic surface storage, an optical disc, or a compact disc read only memory (CD-ROM), and the like. It can also be various devices including one or any combination of the above memories.

[0212] In some embodiments, the executable instructions can be in the form of programs, software, modules, scripts, or code, written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0213] By way of example, the executable instructions can or can not correspond to a file in a file system, can be stored in a part of a file that holds other programs or data, can be stored as one or more scripts or set of instructions in a markup language such as Hyper Text Markup Language (HTML), may

[0214] The above description is only some embodiments of the present application, and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement and improvement within the spirit and scope of the present application shall be included in the protection scope of the present application.

Claims

1. An image processing method, characterized by, The method comprises: acquiring a first image and a second image; extracting identity features of a first target object in the first image to obtain an identity feature vector of the first image; wherein the extracting of the identity features of the first target object in the first image to obtain the identity feature vector of the first image comprises: acquiring image data corresponding to at least two first marks in the first target object, wherein the mark is a point mark in the target object having continuous image data, and the continuous image data refers to a difference between image data corresponding to any point and image data of an adjacent point being small; performing normalization processing on the image data corresponding to the at least two first marks to obtain first normalized data; determining a difference between the first normalized data and an image data mean value as pose information of the first image; and determining the identity feature vector of the first image according to the image data mean value and the pose information of the first image; extracting pose features of a second target object in the second image to obtain a pose feature vector of the second image; obtaining an image synthesis vector corresponding to the first image and the second image based on the identity feature vector and the pose feature vector; generating a synthesized image of the first image and the second image according to the image synthesis vector, wherein a first target object in the synthesized image has a pose of the second target object.

2. The method of claim 1, wherein, The acquiring of the image data corresponding to the at least two first marks in the first target object comprises: determining at least two first regions in the first target object; performing image mark preprocessing on the at least two first regions in the first target object to obtain a first mark of each first region; acquiring the image data corresponding to each first mark.

3. The method of claim 1, wherein, The extracting of the pose features of the second target object in the second image to obtain the pose feature vector of the second image comprises: determining image data corresponding to at least two second marks in the second target object; performing normalization processing on the image data corresponding to the at least two second marks to obtain second normalized data; acquiring an image data mean value of the image data corresponding to the at least two second marks; determining a difference between the second normalized data and the image data mean value as the pose feature vector of the second image.

4. The method of claim 3, wherein, The determining of the image data corresponding to the at least two second marks in the second target object comprises: determining a face image of the second target object; determining at least two second regions related to expression information in the face image; performing image mark preprocessing on the at least two second regions to obtain a second mark of each second region; acquiring the image data corresponding to each second mark.

5. The method according to any one of claims 1 to 4, characterized in that, The obtaining of the image synthesis vector corresponding to the first image and the second image based on the identity feature vector and the pose feature vector comprises: performing normalization processing on the pose feature vector to obtain a normalized feature vector; The normalized feature vector and the identity feature vector are fully connected to obtain the image synthesis vector corresponding to the first image and the second image.

6. The method according to any one of claims 1 to 4, characterized in that, The method further comprises determining a synthesized image of the first image and the second image by using an image processing network. The image processing network is trained by the following steps: inputting a first sample image and a second sample image into the image processing network, determining a first sample label of the first sample image and a second sample label of the second sample image by a label preprocessing model in the image processing network; extracting an identity feature of the first sample image by a target identity encoder in the image processing network to obtain a sample identity feature vector of the first sample image; extracting a feature of the second sample label by a feature extraction layer in the image processing network to obtain a sample pose feature vector of the second sample image; performing instance normalization on the sample identity feature vector and the sample pose feature vector by a generator in the image processing network to obtain a sample synthesis vector corresponding to the first sample image and the second sample image; inputting the sample synthesis vector into a preset loss model to obtain a loss result; correcting network parameters in the label preprocessing model, the target identity encoder and the generator according to the loss result to obtain a corrected image processing network.

7. The method of claim 6, wherein, The method further comprises: before extracting the identity feature of the first sample image by the target identity encoder in the image processing network, inputting the first sample label into the target identity encoder; 8. The method of claim 6, wherein, performing at least one down-sampling process and at least one convolution process on a feature vector corresponding to the first sample label by at least one down-sampling layer and at least one convolution layer in the target identity encoder in sequence to obtain the sample identity feature vector of the first sample image. The method further comprises: before performing the instance normalization on the sample identity feature vector and the sample pose feature vector by the generator in the image processing network, inputting the sample identity feature vector and the sample pose feature vector into the generator; 9. The method of claim 8, wherein, encoding the sample identity feature vector and the sample pose feature vector by an encoder in the generator to obtain an identity encoding vector and a pose encoding vector respectively; performing adaptive instance normalization on the identity encoding vector and the pose encoding vector by an adaptive instance normalization layer in the generator to obtain the sample synthesis vector corresponding to the first sample image and the second sample image. The method further comprises: before extracting the identity feature of the first sample image by the target identity encoder in the image processing network, The feature vector corresponding to the first sample mark is subjected to dimension transformation processing; wherein the dimension of the feature vector after the dimension transformation processing is the same as the vector dimension of the adaptive instance normalization layer.

10. The method of claim 6, wherein, The network parameters in the mark preprocessing model, the target identity encoder and the generator are corrected according to the loss result, to obtain a corrected image processing network. At the Nth network update, a first update gradient of the network parameters is obtained; At the N+1th network update, a second update gradient of the network parameters is obtained; wherein the second update gradient is an update gradient obtained on the basis of the first update gradient; The sum of the first update gradient and the second update gradient is determined as a target update gradient of the network parameters at the N+2th network update; The network parameters in the mark preprocessing model, the target identity encoder and the generator are corrected by using the target update gradient, to obtain the corrected image processing network.

11. An image processing apparatus characterized by comprising: Comprising: The acquisition module is configured to acquire a first image and a second image; The identity feature extraction module is configured to perform identity feature extraction on a first target object in the first image to obtain an identity feature vector of the first image, Wherein, the identity feature extraction module performs identity feature extraction on the first target object in the first image to obtain the identity feature vector of the first image in the following manner: acquiring image data corresponding to at least two first marks in the first target object, wherein the mark is a point mark in the target object having continuous image data, and the continuous image data refers to a small difference between the image data corresponding to any point and the image data of an adjacent point; performing normalization processing on the image data corresponding to at least two first marks to obtain first normalized data; determining a difference between the first normalized data and an image data mean value as pose information of the first image; determining the identity feature vector of the first image according to the image data mean value and the pose information of the first image; The pose feature extraction module is configured to perform pose feature extraction on a second target object in the second image to obtain a pose feature vector of the second image; The processing module is configured to obtain an image synthesis vector corresponding to the first image and the second image based on the identity feature vector and the pose feature vector; The generation module is configured to generate a synthesized image of the first image and the second image according to the image synthesis vector; the first target object in the synthesized image has the pose of the second target object.

12. The apparatus of claim 11, wherein, The identity feature extraction module is further configured to: Determine at least two first regions in the first target object; Perform image mark preprocessing on at least two first regions in the first target object to obtain a first mark of each first region; Acquire the image data corresponding to each first mark.

13. An image processing apparatus characterized by comprising: Comprising: The memory is configured to store executable instructions; A processor configured to implement a method recited in any one of claims 1 to 10 when executing instructions stored in the memory.

14. A computer-readable storage medium, characterized in that, A memory storing executable instructions for causing a processor to implement a method recited in any one of claims 1 to 10 when executing the executable instructions.

15. A computer program product comprising computer programs or instructions, characterized in that, A computer program or instructions which, when executed by a processor, implement a method recited in any one of claims 1 to 10.

Images