Method and device for reconstructing image by using cross-attention

Abstract

Disclosed are a method and a device for reconstructing an image by using cross-attention. The method for reconstructing an image by using cross-attention according to an embodiment of the present invention may comprise the steps of: receiving a plurality of first images including different faces from a user; compressing each of the plurality of first images and expanding the compressed images again to generate a plurality of expanded images for the plurality of first images, respectively; generating a face vector by extracting any one of an identity component and an attribute component from the plurality of first images; and generating a reconstructed image by using cross-attention on the basis of the face vector.

Description

Method and device for reconstructing an image using cross-attention

[0001] The present invention relates to a method for reconstructing an image using cross attention.

[0002] The content described below merely provides background information related to one embodiment of the present invention and does not constitute prior art.

[0003] Face swapping, commonly referred to as deepfake, is a computer vision technique that utilizes visual data and a face synthesis technique that uses human facial data. It involves using two different faces to create a new, synthesized face by blending the features of the other face into the original. It is primarily used in the entertainment and film industries to create fake people, and is also utilized for privacy protection.

[0004] However, existing deepfake models fail to maintain the identity of the source image and fail to properly mimic the pose and attributes of the target image. Furthermore, because face transformation is performed after learning a vast number of source and target images, expensive data collection costs and long training periods are problematic. Therefore, a method that can reflect the characteristics of the target image while maintaining the identity of the source image is required.

[0005] The present invention aims to reconstruct an image using cross attention.

[0006] In addition, the present invention aims to precisely reconstruct an image using images of various resolutions.

[0007] In addition, the present invention aims to generate a natural reconstructed image using cross attention.

[0008] A method for reconstructing an image using cross attention according to one embodiment of the present invention for achieving the above-described object may include the steps of receiving a plurality of first images including different faces from a user, compressing each of the plurality of first images and expanding the compressed images to generate a plurality of expanded images for each of the plurality of first images, generating a face vector by extracting one of an identity element and a characteristic element from the plurality of first images, and generating a reconstructed image using cross attention based on the face vector.

[0009] The plurality of first images may include a source image as a target for extracting an identity element from a face and a target image as a target for extracting a characteristic element from a face, and the step of generating the plurality of extended images may include a step of compressing the plurality of first images based on a U-Net structure, and stepping through a plurality of steps for each of the source images and the target images based on a first artificial intelligence learning model that expands the compressed images again, and a step of stepping through the plurality of steps for each of the source images and the target images based on the first artificial intelligence learning model to generate a plurality of extended images for each step.

[0010] The step of generating a face vector by extracting one of the identity elements and the characteristic elements from the plurality of first images may include the step of extracting the identity element from a face included in the source image using a second artificial intelligence learning model that extracts one of the identity elements and the characteristic elements from an image, the step of extracting the characteristic elements corresponding to a remaining portion excluding the portion corresponding to the identity element from a face included in the target image using the second artificial intelligence learning model, and the step of generating a face vector corresponding to the reconstructed image by mixing the identity elements and the characteristic elements.

[0011] The step of generating the reconstructed image may include a step of generating a second image for each of the plurality of steps by applying cross attention to the plurality of extended images generated for each of the source image and the target image from a lower step among the plurality of steps based on a third artificial intelligence learning model that generates a reconstructed image based on the face vector, and a step of generating a reconstructed image by applying the face vector to each of the second images generated for each of the plurality of steps based on the third artificial intelligence learning model.

[0012] The step of generating a reconstructed image by applying the face vector to the second image may generate the reconstructed image by accumulating the process of applying the face vector to the second image from a lower step among the plurality of steps.

[0013] In addition, a device for reconstructing an image using cross attention according to one embodiment of the present invention may include an image input unit for receiving a plurality of first images including different faces from a user, an extended image generation unit for compressing each of the plurality of first images and re-expanding the compressed images to generate a plurality of extended images for each of the plurality of first images, a face vector generation unit for extracting any one of an identity element and a characteristic element from the plurality of first images to generate a face vector, and a reconstructed image generation unit for generating an image reconstructed using cross attention based on the face vector.

[0014] The plurality of first images include a source image that is a target for extracting an identity element from a face and a target image that is a target for extracting a characteristic element from a face, and the extended image generation unit compresses the plurality of first images based on a U-Net structure, and based on a first artificial intelligence learning model that expands the compressed images again, compresses each of the source images and the target images step-by-step according to a plurality of steps, and based on the first artificial intelligence learning model, expands each of the source images and the target images step-by-step according to the plurality of steps to generate a plurality of extended images for each step.

[0015] The above face vector generation unit may extract an identity element from a face included in the source image using a second artificial intelligence learning model that extracts one of an identity element and a characteristic element from an image, extract a characteristic element corresponding to a remaining area excluding a area corresponding to the identity element from a face included in the target image using the second artificial intelligence learning model, and generate a face vector corresponding to the reconstructed image by mixing the identity element and the characteristic element.

[0016] The above-mentioned reconstructed image generating unit may generate a second image for each of the plurality of stages by applying cross attention to the plurality of extended images generated for each of the source image and the target image from a lower stage among the plurality of stages based on a third artificial intelligence learning model that generates a reconstructed image based on the face vector, and may generate a reconstructed image by applying the face vector to each of the second images generated for each of the plurality of stages based on the third artificial intelligence learning model.

[0017] The above-mentioned reconstructed image generation unit can generate the reconstructed image by accumulating a process of applying the face vector to the second image from a lower stage among the plurality of stages based on the third artificial intelligence learning model.

[0018] According to the present invention, an image can be reconstructed using cross attention.

[0019] In addition, according to the present invention, images can be precisely reconstructed using images of various resolutions.

[0020] Additionally, according to the present invention, a natural reconstructed image can be generated using cross attention.

[0021] FIG. 1 is a flowchart illustrating a method for reconstructing an image using cross attention according to one embodiment of the present invention.

[0022] FIG. 2 is a flowchart illustrating a method for reconstructing an image using cross attention according to one embodiment of the present invention.

[0023] FIG. 3 is a flowchart illustrating a method for reconstructing an image using cross attention according to one embodiment of the present invention.

[0024] FIG. 4 is a flowchart illustrating a method for reconstructing an image using cross attention according to one embodiment of the present invention.

[0025] FIG. 5 is a diagram showing the configuration of a device for reconstructing an image using cross attention according to one embodiment of the present invention.

[0026] FIG. 6 is a diagram illustrating a computer system according to one embodiment of the present invention.

[0027] FIG. 7 is a diagram illustrating a process of generating a face vector by extracting identity elements and characteristic elements according to one embodiment of the present invention.

[0028] The present invention will be described in detail with reference to the attached drawings. Herein, repetitive descriptions, well-known functions that may unnecessarily obscure the gist of the present invention, and detailed descriptions of configurations are omitted. The embodiments of the present invention are provided to more fully explain the present invention to those of ordinary skill in the art. Accordingly, the shapes and sizes of elements in the drawings may be exaggerated for clarity.

[0029] Although "first" or "second" are used to describe various components, these components are not limited by such terms. Such terms may only be used to distinguish one component from another. Accordingly, a first component referred to below may also be a second component within the technical scope of the present invention.

[0030] Throughout the specification, whenever a part is said to "include" a component, this does not mean that it excludes other components, but rather that it may include other components, unless otherwise stated.

[0031] Hereinafter, a preferred embodiment according to the present invention will be described in detail with reference to the attached drawings.

[0032] FIG. 1 is a flowchart illustrating a method for reconstructing an image using cross attention according to one embodiment of the present invention.

[0033] Referring to FIG. 1, a method for reconstructing an image using cross attention according to one embodiment of the present invention may first receive a plurality of first images containing different faces from a user (S110).

[0034] Next, each of the plurality of first images may be compressed and the compressed images may be expanded again to generate a plurality of expanded images for each of the plurality of first images (S120).

[0035] Next, a face vector can be generated by extracting one of the identity elements and characteristic elements from the plurality of first images (S130).

[0036] Here, identity elements can refer to elements that represent a person's unique identity, such as their eyes, nose, mouth, and eyebrows. Furthermore, characteristic elements can refer to any part of a person other than identity elements, such as hair.

[0037] Next, a reconstructed image can be generated using cross attention based on the above face vector (S140).

[0038] Here, the plurality of first images may include a source image, which is a target for extracting identity elements from a face, and a target image, which is a target for extracting characteristic elements from a face.

[0039] For example, a reconstructed image may mean an image that combines the identity elements of the source image with the characteristic elements (external elements) of the target image.

[0040] FIG. 2 is a flowchart illustrating a method for reconstructing an image using cross attention according to one embodiment of the present invention.

[0041] Referring to FIG. 2, a method for reconstructing an image using cross attention according to an embodiment of the present invention first compresses the plurality of first images based on a U-Net structure, and then based on a first artificial intelligence learning model that expands the compressed images again, the source image and the target image can be compressed step by step according to a plurality of steps (S210).

[0042] Next, based on the first artificial intelligence learning model, each of the source image and the target image can be expanded step by step according to the plurality of steps, thereby generating a plurality of expanded images for each step (S220).

[0043] Here, the first artificial intelligence learning model may include an encoder that compresses the input image step by step in a plurality of stages based on a U-Net structure, and a decoder that expands the image compressed through the encoder step by step in a plurality of stages based on the U-Net structure, thereby generating a plurality of expanded images for each stage.

[0044] The U-Net structure belongs to the encoder-decoder based model, and is a model that can extract image features using not only low-dimensional but also high-dimensional information, while simultaneously identifying the exact location.

[0045] Typically, the encoding stage reduces the dimensionality of the input image by increasing the number of channels to capture its features. In the decoding stage, only the low-dimensional encoded information is used to reduce the number of channels and increase the dimensionality to restore the high-dimensional image. However, the dimensionality reduction in the encoding stage results in the loss of detailed location information about image objects. Furthermore, since the decoding stage only utilizes low-dimensional information, it is impossible to recover this lost location information.

[0046] To address these issues, a concatenation method is used, which merges features from each layer of the encoding stage into each layer of the decoding stage. Direct connections between the encoder and decoder layers are called skip connections. The skip connections are placed parallel to each other, and the layers are arranged so that the left and right are symmetrical around the center, resulting in a U-shaped structure, as the name suggests.

[0047] By utilizing this U-Net structure, it is possible to utilize all images converted to various resolutions in the process of compressing and expanding images, and it has the advantage of being able to exhibit high performance in restoring images by directly connecting (skip connecting) each piece of information generated during compression when expanding.

[0048] At this time, the first artificial intelligence learning model can be trained using the Unet loss function.

[0049] The Unet loss function can indicate how much the image generated through compression and decoding via U-Net differs from the original image. Therefore, the first AI learning model can be trained to minimize the Unet loss function, ensuring smooth compression and decoding of the source and target images.

[0050] FIG. 3 is a flowchart illustrating a method for reconstructing an image using cross attention according to one embodiment of the present invention.

[0051] Referring to FIG. 3, a method for reconstructing an image using cross attention according to one embodiment of the present invention first extracts an identity element from a face included in the source image using a second artificial intelligence learning model that extracts one of an identity element and a characteristic element from the image (S310).

[0052] Next, using the second artificial intelligence learning model, characteristic elements corresponding to the remaining parts of the face included in the target image, excluding the part corresponding to the identity element, can be extracted (S320).

[0053] Next, by mixing the identity element and the characteristic element, a face vector corresponding to the reconstructed image can be generated (S330).

[0054] The second artificial intelligence learning model can extract identity elements from the source image using the Deep3dFaceRecon model and extract characteristic elements excluding the identity elements from the target image.

[0055] Referring to Fig. 7, the Deep3dFaceRecon model can be used to extract faces from the source image (Is) and the target image (It) in 3D form, and identity elements (Identity) and feature elements (Expression, Pose) for each can be extracted (Ct, Cs). Next, feature elements (Expression, Pose) can be extracted from the target image and identity elements (Identity) can be extracted from the source image and mixed (C fuse). Through this, a face vector corresponding to a reconstructed image having the identity elements of the source image and the feature elements of the target image can be generated.

[0056] Additionally, the second artificial intelligence learning model can be trained using the ID loss function and the Attribute loss function.

[0057] The ID loss function can indicate the extent to which identity elements differ between the reconstructed image and the source image. Similarly, the attribute loss function can indicate the extent to which feature elements differ between the reconstructed image and the target image. Therefore, the second AI learning model can be trained to minimize the ID loss function and attribute loss function, thereby generating a good reconstructed image.

[0058] In one embodiment, the identity element includes the shape of the face, and the second artificial intelligence learning model can be trained using a Shape loss function.

[0059] The Shape loss function can indicate the degree of facial shape difference between the reconstructed image and the source image. Therefore, the second AI learning model can be trained to minimize the Shape loss function, resulting in better reconstructed images. This allows for more natural-looking reconstructed images.

[0060] FIG. 4 is a flowchart illustrating a method for reconstructing an image using cross attention according to one embodiment of the present invention.

[0061] Referring to FIG. 4, a method for reconstructing an image using cross attention according to an embodiment of the present invention first generates a second image for each of the plurality of steps by applying cross attention to the plurality of extended images generated for each of the source image and the target image, starting from a lower step among the plurality of steps, based on a third artificial intelligence learning model that generates a reconstructed image based on the face vector (S410).

[0062] Here, cross-attention can mean giving additional value to the part that needs more attention for two different input values.

[0063] Next, based on the third artificial intelligence learning model, a reconstructed image can be generated by applying the face vector to each of the second images generated in each of the plurality of steps (S420).

[0064] At this time, the process of applying the face vector to the second image from a lower stage among the plurality of stages can be accumulated to generate the reconstructed image.

[0065] For example, using the third artificial intelligence learning model, cross-attention can be applied to the lowest-dimensional extended images among the multiple extended images generated by the first artificial intelligence learning model to generate a second image, and a face vector can be applied to this to generate a first reconstructed image. Next, cross-attention can be applied to the second lower-dimensional extended images and a face vector can be applied to this to generate a second reconstructed image. In the same manner, the highest-dimensional extended images can be accumulated to generate a final reconstructed image.

[0066] This has the effect of generating a reconstructed image in which elements extracted from the source image and target image are well mixed.

[0067] FIG. 5 is a diagram showing the configuration of a device for reconstructing an image using cross attention according to one embodiment of the present invention.

[0068] Referring to FIG. 5, a device for reconstructing an image using cross attention according to an embodiment of the present invention may include an image input unit for receiving a plurality of first images including different faces from a user, an extended image generation unit for compressing each of the plurality of first images and re-expanding the compressed images to generate a plurality of extended images for each of the plurality of first images, a face vector generation unit for extracting one of an identity element and a characteristic element from the plurality of first images to generate a face vector, and a reconstructed image generation unit for generating an image reconstructed using cross attention based on the face vector.

[0069] At this time, the plurality of first images may include a source image that is a target for extracting identity elements from a face and a target image that is a target for extracting characteristic elements from a face.

[0070] According to one embodiment, the extended image generation unit may compress the plurality of first images based on a U-Net structure, and may stepwise compress each of the source image and the target image in a plurality of steps based on a first artificial intelligence learning model that expands the compressed images again, and may stepwise expand each of the source image and the target image in a plurality of steps based on the first artificial intelligence learning model to generate a plurality of extended images for each step.

[0071] According to one embodiment, the face vector generation unit may extract an identity element from a face included in the source image using a second artificial intelligence learning model that extracts one of an identity element and a characteristic element from an image, extract a characteristic element corresponding to a remaining portion of a face included in the target image excluding a portion corresponding to the identity element using the second artificial intelligence learning model, and generate a face vector corresponding to the reconstructed image by mixing the identity element and the characteristic element.

[0072] According to one embodiment, the reconstructed image generating unit may generate a second image for each of the plurality of steps by applying cross attention to the plurality of extended images generated for each of the source image and the target image from a lower step among the plurality of steps based on a third artificial intelligence learning model that generates a reconstructed image based on the face vector, and may generate a reconstructed image by applying the face vector to each of the second images generated for each of the plurality of steps based on the third artificial intelligence learning model.

[0073] According to one embodiment, the reconstructed image generation unit can generate the reconstructed image by accumulating a process of applying the face vector to the second image from a lower stage among the plurality of stages based on the third artificial intelligence learning model.

[0074] FIG. 6 is a diagram illustrating a computer system according to one embodiment of the present invention.

[0075] A device for reconstructing an image using cross attention according to one embodiment of the present invention can be implemented in a computer system (1000) such as a computer-readable recording medium.

[0076] Referring to FIG. 6, a computer system (1000) may include one or more processors (1010), memory (1030), a user interface input device (1040), a user interface output device (1050), and storage (1060) that communicate with each other via a bus (1020). In addition, the computer system (1000) may further include a network interface (1070) connected to a network (1080). The processor (1010) may be a central processing unit or a semiconductor device that executes processing instructions stored in the memory (1030) or the storage (1060). The memory (1030) and the storage (1060) may be various forms of volatile or non-volatile storage media. For example, the memory may include ROM (1031) or RAM (1032).

[0077] The specific implementations described in the present invention are merely exemplary embodiments and do not limit the scope of the present invention in any way. For the sake of brevity, descriptions of conventional electronic components, control systems, software, and other functional aspects of the systems may be omitted. In addition, the lines connecting or connecting members between components depicted in the drawings are merely exemplary functional connections and / or physical or circuit connections, and may be replaced or represented as various additional functional connections, physical connections, or circuit connections in an actual device. In addition, unless specifically mentioned as “essential,” “important,” etc., a component may not be absolutely necessary for the application of the present invention.

[0078] Therefore, the idea of ​​the present invention should not be limited to the embodiments described above, and not only the scope of the patent claims described below but also all scopes equivalent to or equivalently modified from the scope of the patent claims are considered to fall within the scope of the idea of ​​the present invention.

[0079] According to the method and device for reconstructing an image using cross attention according to embodiments of the present invention, an image can be precisely reconstructed using images of various resolutions, and a natural reconstructed image can be generated using cross attention, so that it has industrial applicability.

Claims

A step of receiving a plurality of first images containing different faces from a user; A step of compressing each of the plurality of first images and expanding the compressed images to generate a plurality of expanded images for each of the plurality of first images; A step of generating a face vector by extracting one of the identity elements and characteristic elements from the plurality of first images; and A step of generating a reconstructed image using cross attention based on the above face vector. A method for reconstructing an image using cross attention, comprising: In the first paragraph, The above plurality of first images are, A source image as a target for extracting identity elements from a face; and Target image, which is the target for extracting characteristic elements from the face Including, The step of generating the above multiple extended images is: A step of compressing the plurality of first images based on the U-Net structure and compressing each of the source image and the target image step by step according to a plurality of steps based on a first artificial intelligence learning model that expands the compressed images again; and A step of generating multiple expanded images for each step by step expanding each of the source image and the target image based on the first artificial intelligence learning model. A method for reconstructing an image using cross attention, comprising: In the second paragraph, The step of generating a face vector by extracting one of the identity elements and characteristic elements from the plurality of first images is as follows: A step of extracting an identity element from a face included in the source image using a second artificial intelligence learning model that extracts either an identity element or a characteristic element from the image; A step of extracting characteristic elements corresponding to the remaining parts of the face included in the target image, excluding the part corresponding to the identity element, using the second artificial intelligence learning model; and A step of generating a face vector corresponding to the reconstructed image by mixing the identity element and the characteristic element. A method for reconstructing an image using cross attention, comprising: In the third paragraph, The step of generating the above reconstructed image is: A step of generating a second image for each of the plurality of stages by applying cross attention to the plurality of extended images generated for each of the source image and the target image, step by step from a lower stage among the plurality of stages, based on a third artificial intelligence learning model that generates an image reconstructed based on the face vector; and A step of generating a reconstructed image by applying the face vector to each of the second images generated for each of the plurality of steps based on the third artificial intelligence learning model. A method for reconstructing an image using cross attention, comprising: In paragraph 4, The step of generating a reconstructed image by applying the face vector to the second image is as follows: A method for reconstructing an image using cross attention, wherein the reconstructed image is generated by accumulating a process of applying the face vector to the second image from a lower stage among the plurality of stages. An image input unit for receiving a plurality of first images containing different faces from a user; An expanded image generation unit that compresses each of the plurality of first images and expands the compressed images to generate a plurality of expanded images for each of the plurality of first images; A face vector generation unit that generates a face vector by extracting one of the identity elements and characteristic elements from the plurality of first images; and A reconstructed image generation unit that generates a reconstructed image using cross attention based on the above face vector. A device for reconstructing an image using cross attention, comprising: In paragraph 6, The above plurality of first images are, A source image as a target for extracting identity elements from a face; and Target image, which is the target for extracting characteristic elements from the face Including, The above extended image generation unit, Compressing the plurality of first images based on the U-Net structure, and compressing the source image and the target image step by step in multiple stages based on the first artificial intelligence learning model that expands the compressed images again, A device for reconstructing an image using cross attention, which generates multiple expanded images for each step by step expanding each of the source image and the target image based on the first artificial intelligence learning model. In paragraph 7, The above face vector generation unit, Extracting identity elements from a face included in the source image using a second artificial intelligence learning model that extracts either identity elements or characteristic elements from the image, Using the second artificial intelligence learning model, characteristic elements corresponding to the remaining parts of the face included in the target image are extracted, excluding the part corresponding to the identity element, A device for reconstructing an image using cross attention, which generates a face vector corresponding to the reconstructed image by mixing the identity element and the characteristic element. In paragraph 8, The above reconstruction image generation unit, Based on a third artificial intelligence learning model that generates an image reconstructed based on the above face vector, cross attention is applied to the plurality of extended images generated for each of the source image and the target image from a lower stage among the plurality of stages to generate a second image for each of the plurality of stages, A device for reconstructing an image using cross attention, which generates a reconstructed image by applying the face vector to each of the second images generated in each of the plurality of steps based on the third artificial intelligence learning model. In paragraph 9, The above reconstruction image generation unit, A device for reconstructing an image using cross attention, which generates the reconstructed image by accumulating a process of applying the face vector to the second image from a lower stage among the plurality of stages based on the third artificial intelligence learning model.

Images