Image providing apparatus and method for providing image

Abstract

The invention relates to an image providing apparatus and method for providing an image. The invention discloses an image providing apparatus and an image providing method. The image providing apparatus includes: a first preprocessor configured to generate a first image by detecting a first face region from each frame of an original video; a second preprocessor configured to generate a second image by detecting a second face region from the original image; and a similar image generator configured to generate similar images respectively corresponding to a gaze position of the viewer based on the first image and the second image.

Description

Technical Field

[0001] Embodiments of the present disclosure generally relate to an image providing apparatus and an image providing method capable of providing a stereoscopic image (eg, a 3D image). Background Art

[0002] Image sensing devices are devices that capture optical images by converting light into electrical signals using photosensitive semiconductor materials that react to light. With the development of the automotive, medical, computer, and communication industries, the demand for high-performance image sensing devices is increasing in various fields such as smartphones, digital cameras, game consoles, IoT (Internet of Things), robots, security cameras, and medical micro cameras.

[0003] In addition, technology has been developed to provide a three-dimensional (3D) image that can express a 3D effect to a viewer (e.g., a user). Recently, technology that can provide a 3D image to a viewer (e.g., a user) without using a separate device (e.g., 3D glasses) has been studied. In particular, technology for realizing parallax as a method of displaying different images according to the gaze position of the viewer has been intensively studied and developed. However, in order to realize this technology, it is necessary to prepare a plurality of images or videos (moving images) according to the gaze position of the viewer, which causes considerable inconvenience in use. Summary of the invention

[0004] According to an embodiment of the present disclosure, an image providing device may include: a first preprocessor configured to generate a first image by detecting a first facial area from each frame of an original video; a second preprocessor configured to generate a second image by detecting a second facial area from the original image; and a similar image generator configured to generate similar images corresponding to the gaze positions of a viewer, respectively, based on the first image and the second image.

[0005] According to another embodiment of the present disclosure, an image providing device may include: a similar image generator, which is configured to generate similar images corresponding to the gaze position of the viewer respectively based on a first image generated using each frame of an original video and a second image generated using the original image; and an image display, which is configured to select a similar image corresponding to the current gaze position of the viewer from the similar images and output the selected similar image on a screen.

[0006] According to another embodiment of the present disclosure, an image providing method may include the following steps: generating a first image using each frame of an original video; generating a second image using the original image; generating similar images corresponding to the gaze positions of the viewer respectively based on the first image and the second image; and selecting a similar image corresponding to the current gaze position of the viewer from the generated similar images to output the selected similar image on a screen. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] The above and other features and advantageous aspects of embodiments of the present disclosure will become readily apparent with reference to the following detailed description when considered in conjunction with the accompanying drawings.

[0008] Figure 1 is a block diagram illustrating an image providing apparatus according to an embodiment of the present disclosure.

[0009] Figure 2 The present invention is shown in the embodiment of the present invention. Figure 1 A flow chart of the operation of the first pre-processor is shown.

[0010] Figure 3 The present invention is shown in the embodiment of the present invention. Figure 1 A flow chart of the operation of the second pre-processor is shown.

[0011] Figure 4 The present invention is shown in the embodiment of the present invention. Figure 1 A conceptual diagram of the operation of the Similarity Image Generator is shown.

[0012] Figure 5 The training according to the embodiment of the present disclosure is shown Figure 4 A conceptual diagram of the method of the first feature extractor, the second feature extractor and the image restoration unit shown.

[0013] Figure 6 The present invention is a diagram showing an embodiment of the present invention. Figure 1 A block diagram of a computing device corresponding to an image providing device. DETAILED DESCRIPTION

[0014] The present disclosure provides an embodiment of an image providing device and an image providing method capable of providing a stereoscopic image (e.g., a 3D image), which can be used in a configuration that substantially solves one or more technical or engineering problems and alleviates limitations or disadvantages encountered in some other image providing devices. Some embodiments of the present disclosure relate to an image providing device and an image providing method capable of providing a three-dimensional (3D) image without the need to prepare multiple images or multiple videos (motion images). Recognizing the above problems, an image providing device and an image providing method according to an embodiment of the present disclosure can provide a 3D image using only one image and one video (motion image) without the need to prepare multiple images or multiple videos (motion images).

[0015] Reference will now be made in detail to embodiments of the present disclosure, examples of which are illustrated in the accompanying drawings. Whenever possible, the same reference numerals will be used throughout the accompanying drawings to represent the same or similar parts. Although the present disclosure is susceptible to various modifications and alternative forms, specific embodiments thereof are illustrated by way of example in the accompanying drawings. However, the present disclosure should not be construed as being limited to the embodiments set forth herein.

[0016] Hereinafter, various embodiments will be described with reference to the accompanying drawings. However, it should be understood that the embodiments of the present disclosure are not limited to specific embodiments, but include various modifications, equivalents and / or substitutions of the embodiments. The embodiments of the present disclosure can provide various effects that can be directly or indirectly recognized.

[0017] Various embodiments of the present disclosure are directed to an image providing apparatus and an image providing method capable of providing a 3D image without preparing a plurality of images or a plurality of videos (moving images).

[0018] It is to be understood that both the foregoing general description and the following detailed description of the present disclosure are exemplary and descriptive and are intended to provide further description of the claimed embodiments.

[0019] Figure 1 is a block diagram showing an image providing apparatus 10 according to an embodiment of the present disclosure.

[0020] Reference Figure 1 The image providing device 10 can be used as a device for providing a 3D image to a viewer. In particular, the image providing device 10 can provide different images according to the gaze position of the viewer. According to one embodiment, the image providing device 10 can be a head mounted display (HMD) or a 3D display.

[0021] The image providing apparatus 10 may include a first preprocessor 100 , a second preprocessor 200 , a head angle information storage section 300 , a similar image generator 400 , and an image display 500 .

[0022] The first preprocessor 100 may detect the first image IMG1 by detecting a face area from each frame of an original video (i.e., an original motion image). The original video may refer to an image including the face of at least one person. In some embodiments, the original video includes the face of one person, but the scope of the present disclosure is not limited thereto. Even when the original video includes facial images of multiple people, the technical concept of the present disclosure is basically applicable (however, in this case, an operation of selecting one of the multiple facial images may be added). The original video may include multiple frames, and the operation of the first preprocessor 100 may be performed sequentially on the multiple frames.

[0023] In addition, the face region of the first image IMG1 may be a region that the first preprocessor 100 crops from each frame of the original video to include a person's face. For example, the face region may have a rectangular shape, but the scope of the present disclosure is not limited thereto, and the face region may have any shape (e.g., a circular shape, a hexagonal shape, etc.).

[0024] In some embodiments, when the frame of the original video corresponds to a face region, the first pre-processor 100 may omit an operation of cropping the face region from the frame of the original video, and may generate the first image IMG1.

[0025] In addition, each frame of the original video may include a color image composed of RGB (red, green, blue) corresponding to color information about a scene and a depth image including information about a distance to the scene.

[0026] The second preprocessor 200 may generate a second image IMG2 by detecting a face region from an original image. The original image may be an image including the face of at least one person. In some embodiments, for convenience of description, the original image includes the face of one person, but the scope of the present disclosure is not limited thereto, and the technical concept of the present disclosure is basically applicable even when the original image includes facial images of multiple people (however, in this case, an operation of selecting one of the multiple facial images may be added).

[0027] In addition, the face region of the second image IMG2 may be a region that the first preprocessor 100 cuts out from the original video to include a person's face. For example, the face region may have a rectangular shape, but the scope of the present disclosure is not limited thereto, and the face region may have any shape (e.g., a circular shape, a hexagonal shape, etc.).

[0028] In some embodiments, when the original image corresponds to a face region, the second pre-processor 200 may omit an operation of cropping the face region from the original image and may generate the second image IMG2.

[0029] In addition, the original image may include a color image composed of RGB (Red, Green, Blue) corresponding to color information about a scene and a depth image including information about a distance to the scene.

[0030] In some embodiments, a face region included in each frame of an original video may be defined as a first face region, and a face region included in an original image may be defined as a second face region.

[0031] Furthermore, a first person corresponding to the first facial region may be different from a second person corresponding to the second facial region, but the scope of the present disclosure is not limited thereto, and it should be noted that the first person and the second person may be the same person.

[0032] The head angle information storage section 300 may store head angle information HA including a head angle corresponding to each of gaze positions of the viewer. The gaze position of the viewer may refer to a relative position of the viewer's head with respect to the screen of the image display 500.

[0033] The head angle may include an angle of rotation around each of a first axis (e.g., X-axis), a second axis (e.g., Y-axis), and a third axis (e.g., Z-axis) that are perpendicular to each other. In some embodiments, the angle obtained when the first axis is used as the rotation axis may be defined as a pitch angle, the angle obtained when the second axis is used as the rotation axis may be defined as a yaw angle, and the angle obtained when the third axis is used as the rotation axis may be defined as a roll angle. That is, the head angle information HA may include a pitch angle, a yaw angle, and a roll angle, which are head angles corresponding to each of the gaze positions of the viewer.

[0034] The similar image generator 400 may generate similar images SI1 to Sin (where "n" is an integer of 2 or more) corresponding to the gaze position of the viewer, respectively, based on the first image IMG1, the second image IMG2, and the head angle information HA. Each of the similar images SI1 to Sin may be an image in which the face of the second image IMG2 has not only a facial expression corresponding to the face of the first image IMG1, but also a head angle of the head angle information HA. Here, the number of similar images SI1 to Sin may be predetermined based on the range that the gaze position of the viewer may have and the number of images to be differently expressed within the range.

[0035] The method of enabling the similar image generator 400 to generate the similar images SI1 to Sin will be referred to later. Figure 4 describe.

[0036] The image display 500 may select a similar image corresponding to the current gaze position of the viewer from the similar images SI1 to Sin generated by the similar image generator 400, and may output the selected similar image on the screen. At this time, the image display 500 may not only output the corresponding similar image on the screen, but also output an image obtained by synthesizing the face area of ​​the original video with the corresponding similar image on the screen. Here, such synthesis may mean replacing the face of the person included in the face area of ​​the original video with the face of the person included in the similar image. This synthesis operation may be performed by the image display 500, but may also be performed by a separate image synthesizer (not shown).

[0037] The image display 500 can obtain the viewer's current gaze position by identifying the relative position of the viewer's gaze position relative to the screen of the image display 500 using a separate sensor included in the image providing device 10 (e.g., a gyro sensor, a pupil sensor, an ultrasonic sensor, an image sensor, etc.).

[0038] exist Figure 1 In the example of , when the current gaze position of the viewer is represented by 510 , the image display 500 may select a first similarity image SI1 (a similarity image corresponding to the current gaze position 510 ), and may output the selected first similarity image SI1 on the screen.

[0039] Alternatively, when the current gaze position of the viewer is represented by 520, the image display 500 may select a third similarity image SI3 (a similarity image corresponding to the current gaze position 520) and output the selected third similarity image SI3 on ​​the screen.

[0040] from Figure 1 As can be seen from the example, when the current gaze position of the viewer is represented by 510, a facial image similar to the front face of the viewer looking straight ahead can be displayed on the screen. When the current gaze position of the viewer is represented by 520, a facial image similar to the left side of the viewer's face can be displayed on the screen instead of the facial image showing the front face of the viewer.

[0041] The image providing device 10 may provide a stereoscopic image (eg, a 3D image) corresponding to the current gaze position of the viewer using only one original video and only one image.

[0042] Figure 2 The present invention is shown in the embodiment of the present invention. Figure 1 A flow chart of the operation of the first pre-processor 100 is shown.

[0043] Reference Figure 2, the first preprocessor 100 may receive the original video OV (in operation S110). The original video OV may be stored in a memory (not shown) located inside the image providing device 10 and then provided to the first preprocessor 100. The original video OV may include a plurality of frames that are sequentially performed in time sequence, and each of the plurality of frames may be distinguished by a frame index (i). For example, the first frame of the original video OV may have a frame index of 0 (i=0), and the second frame may have a frame index that increases from 0 to 1. In some embodiments, the initial value of the frame index (i) may be set to 0.

[0044] The first pre-processor 100 may extract a frame FRi corresponding to a current frame index (i) from the original video OV (in operation S120).

[0045] The first preprocessor 100 may detect the face region FC1 within the frame FRi (at operation S130). In some embodiments, the method of detecting the face region FC1 may be implemented as a method using a face detection algorithm (e.g., a Haar-like feature algorithm). In some other embodiments, the method of detecting the face region FC1 may be implemented as a method of manually selecting the face region FC1 through a user input (e.g., mouse drag) generated by a user of the image providing device 10.

[0046] If the face region FC1 is not detected in the frame FRi (ie, No in operation S140), the first pre-processor 100 may increase the frame index (i) by 1 (i=i+1) and may proceed to the next frame to perform operation S120 again.

[0047] If the face region FC1 is detected in the frame FRi (ie, YES in operation S140), the first pre-processor 100 may extract the face region FC1 from the frame FRi (in operation S150).

[0048] The first preprocessor 100 may adjust the size of the face region FC1 to a predetermined reference size (at operation S160). Here, the reference size may be set to any value within a range smaller than the size of the frame FRi. For example, if the size of the frame FRi is 1920×1080, the reference size may be 100×100. Figure 2 In the example of , for convenience of description, an example of enlarging the size of the face region FC1 to the reference size is shown, but according to another example, the size of the face region FC1 may also be reduced to the reference size. Here, the enlargement of the face region FC1 may be performed using an upscaling technique (e.g., bilinear, bicubic, nearest neighbor, lanczos, spline, etc.), and the reduction of the face region FC1 may be performed using a downscaling technique (e.g., median, mean, etc.).

[0049] The face region adjusted to the reference size may be defined as a first image IMG1.

[0050] If the frame index (i) does not correspond to the last frame of the original video OV (ie, "No" in operation S170), operation S145 may be performed so that operation S120 may be performed again for the next frame.

[0051] If the frame index (i) corresponds to the last frame of the original video OV (ie, 'yes' in operation S170), the operation of the first pre-processor 100 may be terminated.

[0052] Figure 3 The present invention is shown in the embodiment of the present invention. Figure 1 A flow chart of the operation of the second pre-processor 200 is shown.

[0053] Reference Figure 3 The second preprocessor 200 may receive the original image OI (at operation S210 ). The original image OI may be stored in a memory (not shown) located inside the image providing apparatus 10 and then provided to the second preprocessor 200 .

[0054] The second preprocessor 200 may detect the face region FC2 within the original image OI (at operation S220). In some embodiments, the method of detecting the face region FC2 may be implemented as a method using a face detection algorithm (e.g., a Haar feature algorithm). In some other embodiments, the method of detecting the face region FC2 may be implemented as a method of manually selecting the face region FC1 through a user input (e.g., mouse drag) generated by a user of the image providing device 10.

[0055] If the face region FC2 is not detected in the original image OI (ie, 'NO' in operation S230), the second pre-processor 200 may receive another original image OI from a memory (not shown).

[0056] If the face region FC2 is detected in the original image OI (ie, 'YES' in operation S230), the second pre-processor 200 may extract the face region FC2 from the original image OI (in operation S240).

[0057] The second preprocessor 200 may adjust the position of the face included in the face region FC2 to a predetermined reference position (at operation S250). The reference position may refer to a specific position where a straight line interconnecting the centers of the two eyes of the face image coincides with the horizontal direction, and another straight line interconnecting the centers of the nose and mouth of the face coincides with the vertical direction. In some embodiments, the second preprocessor 200 may detect the eyes, nose, and mouth of the face using a contour analysis technique (e.g., wavelet transform). The face region including the face whose position is adjusted to correspond to the reference position may be defined as an adjusted face region FC2'.

[0058] The second preprocessor 200 may adjust the size of the adjusted face region FC2' to a predetermined reference size (in operation S260). Here, the reference size may be Figure 2 The same dimensions as those described in the reference dimensions. Figure 3 In the example of , for convenience of description, an example is shown in which the size of the adjusted face region FC2 ′ is enlarged to the reference size, but according to another example, the size of the adjusted face region FC2 ′ may also be reduced to the reference size.

[0059] The adjusted face region adjusted to the reference size may be defined as a second image IMG2. When the second image IMG2 is generated, the operation of the second preprocessor 200 may be terminated. By adjusting the face region to the reference position and the reference size, the speed and accuracy of subsequent image processing to be performed by the similar image generator may be further improved.

[0060] Figure 4 The present invention is shown in the embodiment of the present invention. Figure 1 A conceptual diagram of the operation of the similarity image generator 400 is shown.

[0061] Reference Figure 4 , the similar image generator 400 may include a first feature extractor 410, a second feature extractor 420, a head rotation unit (ie, a head rotation component) 430, and an image restoration unit 440 (ie, an image restoration component).

[0062] The first feature extractor 410 may extract first feature information FI1 from the first image IMG1 including both the first depth image IMG1_D and the first color image IMG1_C. The first depth image IMG1_D may include information about the distance of each pixel to the scene. The first color image IMG1_C may include color information (red, green, blue) of the scene about each pixel.

[0063] The first feature information FI1 may be information indicating features of a face included in the first image IMG1. In some embodiments, the first feature information FI1 may include landmark information and rotation information.

[0064] The feature point information may include 3D coordinates of facial feature points (e.g., at least one of the nose, mouth, eyes, ears, and chin). Here, the feature point may mean a representative point of each eye, nose, mouth, chin, and ear. For example, the representative point may be the center of each of the nose, eyes, ears, mouth, and chin, but the scope of the present disclosure is not limited thereto.

[0065] The rotation information may include a rotation angle (ie, a pitch angle, a yaw angle, and a roll angle) obtained when the face is rotated about each of a first axis (X axis), a second axis (Y axis), and a third axis (Z axis).

[0066] In some embodiments, the first feature extractor 410 may generate first feature information FI1 from the first image IMG1 using a deep learning model required to analyze features of an image (e.g., a U-net encoder, a self-attention GAN (Generative Adversarial Network) encoder, etc.).

[0067] The second feature extractor 420 may extract second feature information FI2 from the second image IMG2 including both the second depth image IMG2_D and the second color image IMG2_C. The second depth image IMG2_D may include information about the distance of each pixel to the scene. The second color image IMG2_C may include color information (red, green, blue) of the scene about each pixel.

[0068] The second feature information FI2 may be information indicating features of a face included in the second image IMG2. In some embodiments, the second feature information FI2 may include feature point information and rotation information.

[0069] The feature point information may include the 3D coordinates of the feature points of each of the nose, mouth, eyes, ears, and chin of the face. The rotation information may include the rotation angles (i.e., pitch angle, yaw angle, and roll angle) obtained when the face is rotated around each of the first axis (X axis), the second axis (Y axis), and the third axis (Z axis).

[0070] In some embodiments, the second feature extractor 420 may generate second feature information FI2 from the second image IMG2 using a deep learning model (e.g., a U-net encoder, a self-attention GAN encoder, etc.) required to analyze features of the image.

[0071] The head rotation unit 430 may generate the correction feature information FI1' by converting the first feature information FI1 in response to the head angle information HA. In some embodiments, when the face is rotated so that the pitch angle, yaw angle, and roll angle included in the rotation information of the first feature information FI1 are respectively equal to the pitch angle, yaw angle, and roll angle included in the head angle information HA, the head rotation unit 430 may calculate the 3D coordinates of each feature point that changes in response to such rotation.

[0072] For example, Figure 4 As shown, when the face must be rotated counterclockwise around the second axis (Y axis) by a preset angle (αm) so that the pitch angle and roll angle included in the rotation information of the first feature information FI1 become the same as the pitch angle and roll angle included in the head angle information HA, and the yaw angle of the rotation information becomes the same as the yaw angle of the head angle information HA, the head rotation unit 430 can calculate the 3D coordinates of each feature point that changes in response to such rotation by using 3D modeling technology.

[0073] The image restoration unit 440 may generate similar images SI1 to SIn based on the correction feature information FI1 ′, the second feature information FI2 , and the second image IMG2 .

[0074] Each of the similar images SI1 to SIn may correspond to one head angle information HA, and the number of similar images SI1 to SIn may be equal to the number of head angles included in the head angle information HA. That is, for each head angle information HA, the operation of enabling the image restoration unit 440 to generate a similar image may be repeated N times.

[0075] The image restoration unit 440 can correct the facial expression (eg, Figure 4 The mouth shape in the example of ) and the head angle are reflected to the second feature information FI2, and the second feature information FI2 can be restored to a color image by referring to the second image IMG2, resulting in a similar image.

[0076] In some embodiments, the image restoration unit 440 can generate similar images SI1 to SIn based on the corrected feature information FI1', the second feature information FI2, and the second image IMG2 by using a deep learning model required to restore the image from the image features (e.g., a U-net decoder, a self-attention GAN decoder, etc.).

[0077] The image restoration unit 440 may generate a current similar image by referring to a previously created similar image, and thus may improve consistency between similar images.

[0078] Figure 5 The training according to the embodiment of the present disclosure is shown Figure 4 A conceptual diagram of a method of a first feature extractor 410, a second feature extractor 420, and an image restoration unit 440 is shown.

[0079] Reference Figure 5 , when the first feature extractor 410, the second feature extractor 420, and the image restoration unit 440 are implemented as a deep learning model, the learning of the deep learning model may be performed using a first learning image TIMG1 and a second learning image TIMG2 extracted from one video. The first learning image TIMG1 may be a cropped image of a face of a specific person included in one frame of the video, and the second learning image TIMG2 may be a cropped image of a face of a specific person included in another frame of the video. Frames corresponding to the first learning image TIMG1 and the second learning image TIMG2, respectively, may be randomly selected.

[0080] The first feature extractor 410 may extract first learning feature information TFI1 from the first learning image TIMG1 including both the first learning depth image TIMG1_D and the first learning color image TIMG1_C.

[0081] The second feature extractor 420 may extract second learning feature information TFI2 from the second learning image TIMG2 including both the second learning depth image TIMG2_D and the second learning color image TIMG2_C.

[0082] The image restoration unit 440 may generate a learning similarity image TSI based on the first learning feature information TFI1 , the second learning feature information TFI2 , and the second learning image TIMG2 .

[0083] The image restoration unit 440 may reflect the facial expression and head angle shown in the first learning feature information TFI1 to the second learning feature information TFI2, and may restore the second learning feature information TFI2 to a color image by referring to the second learning image TIMG2, resulting in the formation of a similar image TSI.

[0084] In addition, the image restoration unit 440 may generate a learning similarity image TSI by referring to a previously generated learning similarity image.

[0085] As the weights constituting the first feature extractor 410, the second feature extractor 420, and the image restoration unit 440 are changed, the difference between the correct answer image CAI and the learning similar image TSI (in Figure 5 ) can be minimized, and the learning of the deep learning model can be performed. This learning can be considered to be more effective when more learning images and more videos are used as learning data.

[0086] Figure 6 is shown with Figure 1 A block diagram of a computing device 1000 corresponding to an image signal processor.

[0087] Reference Figure 6 , the computing device 1000 may represent a computer program product for executing Figure 1 The image provides an embodiment of the hardware configuration of the operation of the device 10.

[0088] The computing device 1000 may be mounted on a chip independent of a chip on which an image sensing device (or image generating device) is mounted. According to one embodiment, a chip on which an image sensing device is mounted and a chip on which the computing device 1000 is mounted may be implemented in one package (e.g., a multi-chip package (MCP)), but the scope of the present disclosure is not limited thereto.

[0089] The computing device 1000 may include a processor 1010 , a memory 1020 , an input / output interface 1030 , and a communication interface 1040 .

[0090] Processor 1010 can process execution Figure 1 The processor 1010 may be referred to as the image signal processor 100, but the scope of the present disclosure is not limited thereto.

[0091] The memory 1020 may store data and / or instructions required to perform operations of the components 100 to 500 of the image providing apparatus 10, and may be accessed by the processor 1010. For example, the memory 1020 may be a volatile memory (e.g., a dynamic random access memory (DRAM), a static random access memory (SRAM), etc.) or a nonvolatile memory (e.g., a programmable read-only memory (PROM), an erasable PROM (EPROM), etc.), an EEPROM (electrically erasable PROM), a flash memory, etc.).

[0092] That is, a computer program for executing the operation of the image providing apparatus 10 disclosed in this specification is recorded in the memory 1020 and is executed and processed by the processor 1010 , thereby realizing the operation of the image signal processor 100 .

[0093] The input / output interface 1030 is an interface that connects an external input device (eg, keyboard, mouse, touch panel, etc.) and / or an external output device (eg, display) to the processor 1010 to allow transmission and reception of data.

[0094] The communication interface 1040 is a component that can transmit and receive various data with an external device (eg, an application processor, an external memory, etc.), and may be a device that supports wired or wireless communication.

[0095] As apparent from the above description, the image providing apparatus and the image providing method according to the embodiments of the present disclosure can provide a 3D image using only one image and one video (moving image) without preparing a plurality of images or a plurality of videos (moving images).

[0096] The embodiments of the present disclosure may provide various effects that can be recognized directly or indirectly.

[0097] Although a number of exemplary embodiments have been described, it should be understood that modifications and enhancements to the disclosed embodiments and other embodiments are contemplated based on what is described and / or shown in this disclosure. In addition, these embodiments may be combined to form additional embodiments.

[0098] CROSS-REFERENCE TO RELATED APPLICATIONS

[0099] This patent application claims priority to and the benefit of Korean Patent Application No. 10-2023-0154080, filed on November 9, 2023, the disclosure of which is incorporated herein by reference in its entirety.

Claims

1. An image providing device, the image providing device comprising: a first preprocessor that generates a first image by detecting a first face region from each frame of an original video; a second preprocessor that generates a second image by detecting a second face region from the original image; as well as A similarity image generator generates similarity images respectively corresponding to gaze positions of a viewer based on the first image and the second image.

2. The image providing device according to claim 1, wherein: The similar image generator comprises: a first feature extractor that generates first feature information representing features of a face included in the first image; a second feature extractor that generates second feature information representing features of a face included in the second image; a head rotation component that generates correction feature information by converting the first feature information in response to head angle information including head angles respectively corresponding to the gaze positions of the viewer; and An image restoration component generates the similar image based on the correction feature information, the second feature information and the second image.

3. The image providing device according to claim 2, wherein: Each of the first feature information and the second feature information includes: Feature point information, the feature point information including three-dimensional coordinates of each feature point of the face; and Rotation information including a rotation angle obtained when the face is rotated around each of a first axis, a second axis, and a third axis that are perpendicular to each other.

4. The image providing device according to claim 3, wherein: The head rotation component calculates three-dimensional coordinates of the first feature information changed according to a rotation that allows the rotation angle of the first feature information to be equal to the head angle.

5. The image providing device according to claim 3, wherein: The feature points include at least one of a nose, a mouth, an eye, an ear, and a chin.

6. The image providing device according to claim 3, wherein: The rotation angles include a pitch angle, a yaw angle, and a roll angle.

7. The image providing device according to claim 2, wherein: Each of the similar images is an image in which the face of the second image has the facial expression and the head angle included in the first image.

8. The image providing device according to claim 2, wherein: Each of the first feature extractor and the second feature extractor includes a U-net encoder or a self-attention generative adversarial network (GAN) encoder.

9. The image providing device according to claim 2, wherein: The image restoration component is a U-net decoder or a self-attention GAN decoder.

10. The image providing device according to claim 2, wherein: The image restoration component generates the similar image by referring to a previously created similar image.

11. The image providing device according to claim 1, further comprising an image display, wherein: Selecting a similar image corresponding to the current gaze position of the viewer from among the similar images; and The selected similar image is output on the screen.

12. The image providing device according to claim 11, wherein: The image display: obtaining a composite image by replacing a face included in the first face region of the original video with a face included in the selected similar image; and The composite image is output on the screen.

13. The image providing device according to claim 1, wherein: The first preprocessor generates the first image by cropping the first face region from each frame of the original video.

14. The image providing device according to claim 1, wherein: The second preprocessor generates the second image by cropping the second face region from the original image.

15. The image providing device according to claim 1, wherein: The first preprocessor generates the first image by adjusting the first face region to a predetermined reference size.

16. The image providing device according to claim 15, wherein: The second preprocessor: adjusting the position of the face included in the second face area to a predetermined reference position; and The adjusted second facial region is adjusted to the reference size to generate the second image.

17. An image providing device, the image providing device comprising: a similar image generator that generates similar images corresponding to gaze positions of a viewer, respectively, based on a first image generated using each frame of an original video and a second image generated using the original image; as well as An image display selects a similar image corresponding to a current gaze position of the viewer from among the similar images and outputs the selected similar image on a screen.

18. The image providing device according to claim 17, wherein: The similar image generator comprises: a first feature extractor that generates first feature information representing features of a face included in the first image; a second feature extractor that generates second feature information representing features of a face included in the second image; a head rotation component that generates correction feature information by converting the first feature information in response to head angle information including head angles respectively corresponding to the gaze positions of the viewer; and An image restoration component generates the similar image based on the correction feature information, the second feature information and the second image.

19. The image providing device according to claim 18, wherein: Each of the first feature information and the second feature information includes: Feature point information, the feature point information including three-dimensional coordinates of each feature point of the face; and Rotation information including a rotation angle obtained when the face is rotated around each of a first axis, a second axis, and a third axis that are perpendicular to each other.

20. The image providing device according to claim 19, wherein: The head rotating assembly: The three-dimensional coordinates of the first feature information changed according to the rotation that allows the rotation angle of the first feature information to be equal to the head angle are calculated.

21. The image providing device according to claim 18, wherein: Each of the similar images is an image in which the face of the second image has the facial expression and the head angle included in the first image.

22. A method for providing an image, the method comprising the following steps: Using each frame of the original video to generate a first image; generating a second image using the original image; generating similar images respectively corresponding to gaze positions of the viewer based on the first image and the second image; as well as A similarity image corresponding to the current gaze position of the viewer is selected from among the generated similarity images, so as to output the selected similarity image on a screen.

23. The image providing method according to claim 22, wherein: The step of generating the first image includes the step of extracting a next frame when a face region is not detected in any frame.

24. The image providing method according to claim 22, wherein: The step of generating the second image includes the step of receiving another original image when the face area is not detected in the original image.

Images