Avatar service providing method and system

Abstract

To provide a method of providing an avatar service.SOLUTION: A method of providing an avatar service is provided, comprising: receiving an image including a first person object from a user terminal via an instant messaging application; extracting skeleton information of the first person object; identifying a first user account of the instant messaging application associated with the first person object; and removing the first person object from the image to create a background image.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present disclosure relates to an avatar service providing method and system, and more specifically, to an avatar service providing method and system for generating a synthetic image using an avatar.

Background Art

[0002] The popularity of terminals such as smartphones, tablet PCs, laptop computers, and desktop computers equipped with cameras is expanding. In such an environment, the use of cameras attached to terminals is increasing. As a typical example, a service using an avatar, which is a character that plays the role of the user in a virtual space using the camera attached to the terminal, is provided.

[0003] Although various techniques for replacing the shape of a person included in a photographed image with an avatar are provided, there are inconveniences such as the user having to select an avatar and directly set the pose of the avatar. In particular, naturally synthesizing an avatar similar to the shape of a person included in a photographed image is not an easy task for general users.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] The present disclosure provides an avatar service providing method, a computer program, and an apparatus (system) for solving the above problems.

[0006] An avatar service providing method and apparatus are provided that can naturally generate an avatar taking a pose identical or similar to a person object based on information extracted from the person object included in an image.

Means for Solving the Problem

[0007] The present disclosure can be implemented in various ways including a method, an apparatus (system), or a computer program.

[0008] According to an embodiment of the present disclosure, an avatar service providing method performed by at least one processor includes receiving an image including a first person object from a user terminal via an instant messaging application, extracting skeleton information of the first person object, detecting a first user account of the instant messaging application associated with the first person object, and generating a background image by removing the first person object from the image.

[0009] A computer program for executing the above-described avatar service providing method according to an embodiment of the present disclosure is provided.

[0010] An avatar service providing system according to an embodiment of the present disclosure includes a communication module configured to receive an image including a person object from a user terminal via an instant messaging application, a memory, and at least one processor coupled to the memory and configured to execute at least one computer-readable program included in the memory. The at least one program includes instructions for extracting skeleton information of the person object, detecting a user account of the instant messaging application associated with the person object, and generating a background image by removing the person object from the image.

[0011] A method for providing an avatar service performed by at least one processor according to an embodiment of the present disclosure includes receiving an image including a person object, transferring the image to an external device via an instant messaging application, receiving skeletal information of the person object from the external device, receiving avatar information related to the person object from the external device, receiving a background image with the person object removed from the external device, and generating a composite image in which the person object is replaced with an avatar based on the skeletal information, the avatar information, and the background image.

[0012] A method for providing an avatar service performed by at least one processor according to an embodiment of the present disclosure includes receiving an image including a person object, transferring at least a part of the image to an external device via an instant messaging application, receiving avatar information related to the person object from the external device, extracting skeletal information of the person object, generating a background image by removing the person object from the image, and generating a composite image in which the person object is replaced with an avatar based on the received avatar information, skeletal information, and background image.

Brief Description of the Drawings

[0013] Embodiments of the present disclosure are described with reference to the accompanying drawings described below, where like reference numerals indicate like elements, but are not limited thereto.

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

DETAILED DESCRIPTION OF THE INVENTION

[0014] <SUMMARY OF THE INVENTION> According to an embodiment, it further includes the step of generating a composite image in which a first human object is replaced with an avatar based on first avatar information and the skeletal information of the first human object associated with a first user account.

[0015] According to an embodiment, it further includes the step of transferring the first avatar information, the skeletal information of the first human object, and the background image associated with the first user account to the user terminal.

[0016] According to one embodiment, the step of removing the first person object from the image and generating a background image includes a step of changing pixel values in an area corresponding to the first person object based on pixel values in an area other than the area corresponding to the first person object in the image.

[0017] According to one embodiment, the step of detecting a first user account of an instant messaging application associated with the first person object includes a step of comparing a facial region in the first person object with facial information of a user account of an instant messaging application associated with the user terminal.

[0018] According to one embodiment, facial information of a user account associated with the user terminal is generated based on at least one of at least one image included in profile information of the user account associated with the user terminal or at least one video call image received from the user terminal.

[0019] According to one embodiment, the step of detecting a first user account of an instant messaging application associated with the first person object includes a step of comparing a facial area within the first person object with facial information of user accounts that are friends with the user account of the instant messaging application associated with the user terminal.

[0020] According to one embodiment, the step of generating a composite image includes the steps of converting skeletal information of the first person object into avatar skeletal information based on the first avatar information, generating an avatar image based on the avatar skeletal information and the first avatar information, and inserting the avatar image into a background image.

[0021] According to one embodiment, the method further includes, in response to determining that no avatar information associated with the first user account exists, searching for an avatar that has the highest similarity to the first person object.

[0022] According to one embodiment, the image includes shooting angle information, and the avatar service providing method further includes generating a composite image in which a first person object is replaced by an avatar based on first avatar information related to a first user account, skeleton information of the first person object, and the shooting angle information.

[0023] According to one embodiment, the method further includes estimating the shooting angle information of the image, and generating a composite image in which a first person object is replaced by an avatar based on first avatar information related to a first user account, skeleton information of the first person object, and the shooting angle information.

[0024] According to one embodiment, the image further includes a second person object, and the avatar service providing method further includes extracting skeleton information of the second person object, detecting a second user account of an instant messaging application related to the second person object, and determining a depth difference between the first person object and the second person object in the image.

[0025] According to one embodiment, the step of generating a background image by removing the first person object from the image includes generating a background image by removing the second person object from the image, and the step of generating a composite image includes generating a composite image based on the first avatar information, second avatar information related to the second user account, skeleton information of the first person object, skeleton information of the second person object, and the depth difference.

[0026] According to one embodiment, the step of determining the depth difference between the first person object and the second person object in the image includes at least one of the steps of comparing the foot positions of the first person object and the second person object, comparing the face sizes of the first person object and the second person object, or comparing the depth information of the first person object and the depth information of the second person object based on the depth information included in the image.

[0027] According to one embodiment, the step of detecting the first user account of the instant messaging application associated with the first person object includes the step of transferring, via the instant messaging application, a message requesting information about the user account associated with the first person object to the user terminal in response to determining that the first person object is at least one of a lateral or rear posture of a person.

[0028] According to one embodiment, the step of detecting the first user account of the instant messaging application associated with the first person object includes the step of transferring the first user account to the user terminal via the instant messaging application and the step of receiving, via the instant messaging application, a user account different from the first user account from the user terminal.

[0029] According to one embodiment, the step of extracting the skeleton information of the first person object includes the step of determining whether the size of the region corresponding to the first person object is greater than or equal to a preset threshold.

[0030] In various embodiments of the present disclosure, since the user account is detected in the image captured through face recognition and the avatar associated with the detected user account is used, a composite image can be generated without inputs such as information about the person captured by the user, information about the avatar, etc.

[0031] In various embodiments of the present disclosure, in order to adaptively remove a person object based on background content excluding the person object, an avatar having body proportions different from those of an actual person can be inserted to generate a natural composite image.

[0032] In various embodiments of the present disclosure, since the skeletal information of the person object is changed according to an avatar having different body proportions from an actual person, an image of an avatar taking a pose identical or similar to that of the person object can be generated.

[0033] In various embodiments of the present disclosure, in order to extract the skeletal information of a person object only when the size of the region corresponding to the person object is equal to or greater than a preset threshold, additional processing time (e.g., skeletal information extraction, related user account detection, etc.) caused by an unintended person object included in the person objects included in the image (e.g., a person in the background) can be prevented.

[0034] In various embodiments of the present disclosure, the accuracy of face recognition can be improved by re-training a face recognition model using information input by a user (e.g., user account information related to a person object in an image).

[0035] In various embodiments of the present disclosure, an avatar image reflecting perspective, distortion, size difference, etc. due to a difference in shooting angle can be generated, and accordingly, an avatar image more similar to the pose of the photographed person object can be generated.

[0036] In various embodiments of the present disclosure, when two or more person objects are detected in an image, a composite image reflecting the front and rear positions of the persons can be generated.

[0037] The effects of the present disclosure are not limited to the effects mentioned above, and other effects not mentioned should be clearly understood by a person having ordinary skill in the art to which the present disclosure pertains (referred to as a "person skilled in the art") from the description in the claims.

[0038] <Detailed Description of the Invention> Hereinafter, specific details for implementing the present disclosure will be described in detail with reference to the accompanying drawings, however, in the following description, detailed descriptions of well-known functions and configurations will be omitted if they may unnecessarily obscure the gist of the present disclosure.

[0039] In the accompanying drawings, identical or corresponding components are denoted by the same reference numerals. In addition, in the following description of the embodiments, duplicated descriptions of identical or corresponding components may be omitted. However, omission of a description of a component does not mean that such a component is not included in a certain embodiment.

[0040] The advantages and features of the disclosed embodiments, and methods for achieving them, will become apparent from the following detailed description of the embodiments taken in conjunction with the accompanying drawings. However, the present disclosure is not limited to the embodiments disclosed below, and may be embodied in various different forms. The embodiments are provided solely so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.

[0041] The terms used in this specification will be briefly explained, and the disclosed embodiments will be specifically described. The terms used in this specification are currently commonly used and general terms that have been selected as much as possible while taking into consideration the functions of the present disclosure. However, these terms may change depending on the intentions of engineers in the relevant field, legal precedents, the emergence of new technologies, etc. In addition, in certain cases, the applicant may arbitrarily select terms, and in such cases, their meanings will be described in detail in the relevant description of the invention. Therefore, the terms used in this disclosure should be defined based on the meanings of the terms and the overall content of the present disclosure, rather than simply by the names of the terms.

[0042] In this specification, the singular expression includes the plural expression unless the context clearly dictates otherwise. Furthermore, the plural expression includes the singular expression unless the context clearly dictates otherwise. Throughout the specification, when a part "comprises" a certain element, this does not mean that other elements are excluded, but that other elements may also be included, unless otherwise specified.

[0043] Additionally, the terms "module" or "unit" as used herein refer to a software or hardware component, and the "module" or "unit" performs a certain function. However, the term "module" or "unit" is not limited to software or hardware. A "module" or "unit" may be configured to reside on an addressable storage medium or to execute on one or more processors. Thus, by way of example, a "module" or "unit" may include components such as software components, object-oriented software components, class components, and task components, as well as at least one of a process, function, attribute, procedure, subroutine, program code segment, driver, firmware, microcode, circuit, data, database, data structure, table, array, or variable. Components and "modules" or "units" may be combined into fewer components and "modules" or "units," or further separated into additional components and "modules" or "units."

[0044] According to an embodiment of the present disclosure, a "module" or "unit" can be implemented by a processor and a memory. The "processor" should be broadly interpreted to include general-purpose processors, central processing units (CPUs), microprocessors, digital signal processors (DSPs), controllers, microcontrollers, state machines, etc. In some environments, the "processor" can also refer to application-specific integrated circuits (ASICs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), etc. The "processor" can also refer to a combination of processing devices, such as a combination of a DSP and a microprocessor, a combination of multiple microprocessors, a combination of one or more microprocessors coupled with a DSP core, or any other such configuration combination. Also, the "memory" should be broadly interpreted to include any electronic component capable of storing electronic information. The "memory" can also refer to various types of processor-readable media, such as random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable PROM (EEPROM), flash memory, magnetic or optical data storage devices, registers, etc. If the processor can read information from the memory / or record information in the memory, the memory is said to be in electronic communication with the processor. Memory integrated with the processor is in electronic communication with the processor.

[0045] In the present disclosure, a "user account" can indicate an account generated and used by a user in an instant messaging application or data related thereto. Also, a user account of an instant messaging application can refer to a user who uses the instant messaging application. Similarly, a user who uses an instant messaging or a chat room capable of instant messaging can refer to a user account of the instant application. Also, a user account can include one or more user accounts.

[0046] In the present disclosure, "skeletal information" can indicate information that can simply represent the shape of a human object using straight lines and curves. A straight line can be information that represents a straight part such as an arm / leg from one joint part to another joint part, or from a joint part to the end part of the arm / leg, and a curve can be information that represents a round part such as a head. In one embodiment, circles, ellipses, polygons, etc. can also be used instead of straight lines and curves.

[0047] In the present disclosure, a "background image" can be an image in which human objects in the image are adaptively removed. In one embodiment, only human objects that meet predetermined requirements among the human objects in the image can be removed. Therefore, the background image may include human objects (for example, human objects smaller than a predetermined size).

[0048] FIG. 1 is a drawing showing an example in which a user terminal (120) generates a composite image (140) in which a person object is replaced with an avatar image (150) according to an embodiment of the present disclosure. As shown, a first user (110) can use the user terminal (120) to take a photo of a second user (130). The user terminal (120) can replace the person object with the avatar image (150) of the second user in the captured image that includes the second user (130) as a person object.

[0049] In one embodiment, the user terminal (120) can recognize the shape of the second user (130) included in the captured image as a person object. Whether an object in the image is a person object can be determined by detecting the contour line of the object and then using the shape of the contour line. For example, the user terminal (120) can use a database storing the shapes of a plurality of contour lines corresponding to person objects, or a person object recognition module generated through machine learning or the like. Thereafter, the user terminal (120) can extract the skeleton information of the person object recognized in the captured image.

[0050] The user terminal (120) can detect a user account associated with a person object recognized in an image captured via face recognition. For example, the user account can be a user account used in an instant messaging application. Specifically, the user terminal (120) can compare the face region within the recognized person object with the face information of a plurality of user accounts stored in a database. For example, the user terminal (120) can compare the face region within the person object with the face information of a plurality of user accounts stored in the database, including the face information of the user account associated with the user terminal (120) that captured the image and the face information of the user accounts that are friends with the user account associated with the user terminal (120). In response to detecting a user account associated with the recognized person object, the user terminal (120) can obtain avatar information (e.g., avatar information of the representative avatar) associated with the detected user account.

[0051] In one embodiment, the face information of the user account can be generated based on an image included in the profile information of each user account. Also, an image including a face similar to the image included in the profile information of the user account can be searched for and used to generate the face information of the corresponding user account. For example, an image including a face similar to the image included in the profile information of a corresponding user account can be searched for in images, videos, etc. transferred within a chat room including a specific user account. Additionally or alternatively, the face information of the user account can be generated based on images, videos, etc. transferred by each user account via an instant messaging application. Additionally or alternatively, the face information of the user account can be generated based on video call images transferred by each user account via an instant messaging application.

[0052] In one embodiment, the user terminal (120) can generate a background image with the person object recognized in the captured image adaptively removed. Specifically, the user terminal (120) can remove the person object by changing the pixel values within the area corresponding to the person object based on the pixel values within the area other than the area corresponding to the person object in the captured image. For example, the user terminal (120) can generate a fake image in which the person object is adaptively removed from the captured image by using an image conversion model based on GAN (Generative Advertisarial Network) and the like.

[0053] After that, the user terminal (120) can generate a composite image (140) by using the avatar information, skeleton information, and background image related to the detected user account. Specifically, the user terminal (120) can generate an avatar image (150) that takes the same or a similar pose as the second user (130) by using the skeleton information, and insert the generated avatar image (150) into the background image to generate a composite image (140).

[0054] In one embodiment, the user terminal (120) can generate a composite image (140) when the first user (110) captures an image, a video, etc. by using the in-app camera function within the instant messaging application. Additionally or alternatively, the user terminal (120) can generate a composite image (140) when the first user (110) transfers an image, a video, etc. via the instant messaging application. Additionally or alternatively, the user terminal (120) can generate a composite image (140) when the first user (110) makes a video call via the instant messaging application.

[0055] In various embodiments of the present disclosure, a user account is detected from a captured image through facial recognition and an avatar associated with the detected user account is used, thereby generating a composite image (140) without user input of information about the person captured in the image or the avatar. Furthermore, adaptively removing a person object based on background content that excludes the person object allows a natural-looking composite image (140) to be generated even when an avatar image (150) with body proportions different from that of a real person is inserted. Furthermore, by synthesizing an avatar based on skeletal information of a person object, an avatar posing the same or similar to the captured person can be easily generated without user input.

[0056] Although the above description describes that the user terminal 120 performs all of the above functions, this is not intended to be limiting. For example, some or all of the above functions may be performed by other external devices (e.g., an instant messaging service providing server, an avatar server, a face recognition server, an avatar synthesis server, etc.).

[0057] FIG. 2 is a schematic diagram illustrating a configuration in which an information processing system (200) is communicatively connected to a plurality of user terminals (220_1, 220_2, 220_3) for generating a composite image in which a person object is replaced with an avatar, according to an embodiment of the present disclosure. The information processing system (200) may include a system capable of providing an instant messaging service, including an avatar synthesis service, via a network (210). In one embodiment, the information processing system (200) may include one or more server devices and / or databases, or one or more distributed computing devices and / or distributed databases based on a cloud computing service, capable of storing, providing, and executing computer-executable programs (e.g., downloadable applications) and data related to the instant messaging service and the generation of a composite image in which a person object is replaced with an avatar. The instant messaging service provided by the information processing system (200) may be provided to users via an instant messaging application installed on each of the plurality of user terminals (220_1, 220_2, 220_3). For example, instant messaging services may include text messaging services, video calling services, voice calling services, video streaming services, avatar synthesis services, and the like between users of instant messaging applications.

[0058] A plurality of user terminals (220_1, 220_2, 220_3) can communicate with an information processing system (200) via a network (210). The network (210) can be configured to enable communication between the plurality of user terminals (220_1, 220_2, 220_3) and the information processing system (200). The network (210) can be composed of a wired network such as Ethernet, a wired home network (Power Line Communication), a telephone line communication device, and RS-serial communication, a mobile communication network, a wireless network such as WLAN (Wireless LAN), Wi-Fi, Bluetooth, and ZigBee, or a combination thereof, depending on the installation environment. The communication method is not limited, and it includes not only a communication method that utilizes a communication network (for example, a mobile communication network, a wired Internet, a wireless Internet, a broadcast network, a satellite network, etc.) that the network (210) can include, but also short-range wireless communication between the user terminals (220_1, 220_2, 220_3).

[0059] 2 illustrates a mobile phone terminal 220_1, a tablet terminal 220_2, and a PC terminal 220_3 as examples of user terminals, but the user terminals 220_1, 220_2, and 220_3 may be any computing device capable of wired and / or wireless communication and capable of installing and running an instant messaging application. For example, the user terminal may include a smartphone, a mobile phone, a navigation system, a computer, a laptop, a digital broadcasting terminal, a personal digital assistant (PDA), a portable multimedia player (PMP), a tablet PC, a game console, a wearable device, an internet of things (IoT) device, a virtual reality (VR) device, an augmented reality (AR) device, etc. Also, while FIG. 2 illustrates three user terminals (220_1, 220_2, 220_3) communicating with the information processing system (200) via the network (210), this is not limited thereto, and a different number of user terminals may be configured to communicate with the information processing system (200) via the network (210).

[0060] In one embodiment, the information processing system 200 may generate a composite image in which a human object in an image is replaced with an avatar via an instant messaging application running on the user terminals 220_1, 220_2, and 220_3. If a user account associated with the user terminal does not have an avatar, the information processing system 200 may search for an avatar that is most similar to a human object included in the image and use the searched avatar to generate a composite image in which the human object is replaced with an avatar. If a user account has multiple avatars, the information processing system 200 may provide an avatar synthesis service using a representative avatar of the user account. Alternatively, if a user account has multiple avatars, the information processing system 200 may request the user to select an avatar to use in the avatar synthesis service.

[0061] FIG. 3 is a block diagram showing the internal configuration of a user terminal (220) and an information processing system (200) according to an embodiment of the present disclosure. The user terminal (220) can refer to any computing device on which an instant messaging application can be executed and which can perform wired / wireless communication. For example, it can include the mobile phone terminal (220_1), the tablet terminal (220_2), the PC terminal (220_3) of FIG. 2, etc. As shown in the figure, the user terminal (220) can include a memory (312), a processor (314), a communication module (316), and an input / output interface (318). Similarly, the information processing system (200) can include a memory (332), a processor (334), a communication module (336), and an input / output interface (338). As shown in FIG. 3, the user terminal (220) and the information processing system (200) can be configured to communicate information and / or data via the network (210) using their respective communication modules (316, 336). Also, the input / output device (320) can be configured to input information and / or data to the user terminal (220) via the input / output interface (318), or output information and / or data generated from the user terminal (220).

[0062] The memory (312, 332) can include any non-transitory computer-readable recording medium. According to one embodiment, the memory (312, 332) can include a non-volatile mass storage device such as a RAM (random access memory), ROM (read only memory), disk drive, SSD (solid state drive), flash memory, etc. As another example, non-volatile mass storage devices such as ROM, SSD, flash memory, disk drive, etc. can be included in the user terminal (220) or the information processing system (200) as a separate permanent storage device distinct from the memory. Also, the memory (312, 332) can store an operating system and at least one program code (for example, code for an instant messaging application installed and driven on the user terminal (220)).

[0063] Such software components can be loaded from a computer-readable recording medium separate from the memory (312, 332). Such a separate computer-readable recording medium can include a recording medium directly connectable to these user terminals (220) and information processing systems (200), and can include, for example, computer-readable recording media such as a floppy drive, disk, tape, DVD / CD-ROM drive, memory card, etc. As another example, software components can also be loaded into the memory (312, 332) via a communication module instead of a computer-readable recording medium. For example, at least one program can be loaded into the memory (312, 332) based on a computer program (for example, an application providing an instant messaging application service) installed by a file provided by a file distribution system that distributes developer or application installation files via the network (210).

[0064] The processors (314, 334) may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input / output operations. The instructions can be provided to the processors (314, 334) by a memory (312, 332) or a communication module (316, 336). For example, the processors (314, 334) may be configured to execute instructions received based on program code stored in a recording device such as the memory (312, 332).

[0065] The communication modules (316, 336) can provide a configuration or function for the user terminal (220) and the information processing system (200) to communicate with each other via the network (210), and can provide a configuration or function for the user terminal (220) and / or the information processing system (200) to communicate with other user terminals or other systems (e.g., a separate cloud system, etc.). As an example, a request or data (e.g., an avatar synthesis request, skeleton information extracted from a human object, a background image with the human object removed, a synthesized image with the human object replaced by an avatar, etc.) generated by the processor (314) of the user terminal (220) based on program code stored in a recording device such as the memory (312) can be transmitted to the information processing system (200) via the network (210) under the control of the communication module (316). Conversely, a control signal or instruction provided based on the control of the processor (334) of the information processing system (200) can be received by the user terminal (220) via the communication module (336) of the information processing system (200), the network (210), and the communication module (316) of the user terminal (220). For example, the user terminal (220) can receive avatar information related to a user account detected via the communication module (316) from the information processing system (200), skeleton information extracted from a human object, a background image with the human object removed, a synthesized image with the human object replaced by an avatar, etc.

[0066] The input / output interface 318 may be a means for interfacing with the input / output device 320. For example, the input device may include a camera including an audio sensor and / or an image sensor, a keyboard, a microphone, a mouse, etc., and the output device may include a display, a speaker, a haptic feedback device, etc. As another example, the input / output interface 318 may be a means for interfacing with a device that integrates input and output functions, such as a touchscreen. For example, when the processor 314 of the user terminal 220 processes instructions of a computer program loaded into the memory 312, a service screen or an image of a user's avatar, which is constructed using information and / or data provided by the information processing system 200 or another user terminal, may be displayed on the display via the input / output interface 318. While the input / output device 320 is illustrated as not being included in the user terminal 220 in FIG. 3, this is not limiting and the input / output device 320 may be configured as a single device with the user terminal 220. In addition, the input / output interface 338 of the information processing system 200 may be a means for interfacing with an input or output device (not shown) that may be connected to the information processing system 200 or may be included in the information processing system 200. In FIG. 3, the input / output interfaces 318, 338 are illustrated as elements configured separately from the processors 314, 334, but are not limited thereto, and the input / output interfaces 318, 338 may be configured to be included in the processors 314, 334.

[0067] The user terminal (220) and the information processing system (200) can include more components than those shown in FIG. 3. However, there is no need to clearly illustrate most of the conventional components. According to one embodiment, the user terminal (220) can be implemented to include at least a part of the input / output devices (320) described above. In addition, the user terminal (220) can further include other components such as a transceiver, a GPS (Global Positioning System) module, a camera, various sensors, a database, etc. For example, when the user terminal (220) is a smartphone, it can include components generally included in a smartphone. For example, various components such as an acceleration sensor, a gyro sensor, a camera module, various physical buttons, buttons using a touch panel, input / output ports, and a vibrator for vibration can be implemented to be further included in the user terminal (220).

[0068] According to one embodiment, the processor (314) of the user terminal (220) can be configured to operate an instant messaging application or a web browser application that provides an instant messaging service including an avatar synthesis image generation service. At this time, the program code related to the corresponding application can be loaded into the memory (312) of the user terminal (220). While the application is operating, the processor (314) of the user terminal (220) can receive information and / or data provided from the input / output device (320) and / or received from the information processing system (200) via the communication module (316) through the input / output interface (318), and can process the received information and / or data and store it in the memory (312). Also, such information and / or data can be provided to the information processing system (200) via the communication module (316).

[0069] While the instant messaging application is operating, the processor (314) can receive or select text, images, videos, etc. input via an input device including a touch screen, keyboard, audio sensor and / or camera, microphone, etc. connected to the input / output interface (318), and can store the received text, images and / or videos, etc. in the memory (312), or provide them to the information processing system (200) via the communication module (316) and the network (210). In one embodiment, the processor (314) can provide the captured image received via the input device to the information processing system (200) via the network (210) and the communication module (316). Alternatively, the processor (314) can extract the skeletal information of the human object included in the captured image, generate a background image with the human object removed, and then receive the avatar information of the user account related to the human object included in the captured image from the information processing system (200) to generate a composite image.

[0070] The processor (334) of the information processing system (200) can be configured to manage, process and / or store information and / or data received from a plurality of user terminals and / or a plurality of external systems. In one embodiment, the processor (334) can detect a user account related to the human object included in the image based on the captured image received from the user terminal (220), and search for the avatar information related to the detected user account. Also, a background image with the human object removed from the image can be generated, and the skeletal information of the human object can be extracted. In one embodiment, the processor (334) can generate a composite image in which the human object is replaced by the avatar based on the avatar information, skeletal information and background image with the human object removed related to the user account.

[0071] FIG. 4 is a flowchart showing an avatar service providing method (400) according to an embodiment of the present disclosure. In one embodiment, the avatar service providing method (400) can be performed by an information processing system (e.g., a processor of the information processing system). In other embodiments, the avatar service providing method (400) can be performed by a user terminal (e.g., a processor of the user terminal). For example, when a database storing face information of one or more user accounts and avatar information related to the user account are stored in the user terminal, the user terminal can perform all steps of the avatar service providing method (400). In still other embodiments, the information processing system (e.g., a processor of the information processing system) and the user terminal (e.g., a processor of the user terminal) can perform the steps of the avatar service providing method (400) separately.

[0072] As illustrated, the avatar service providing method (400) can be disclosed by a processor receiving an image including a person object (S410). For example, the processor can receive an image including a person object from the user terminal via an instant messaging application operating on the user terminal. Alternatively, the processor can receive an image including a person object from an image sensor.

[0073] In response to receiving the image, the processor may extract skeletal information of a person object included in the image (S420). In one embodiment, the processor may determine whether the size of an area corresponding to the person object is equal to or greater than a predetermined threshold, and extract skeletal information of the person object only if the size of the area corresponding to the person object is equal to or greater than the threshold. If multiple person objects are recognized from the image, it may be determined whether the size of the area corresponding to each object is equal to or greater than a predetermined threshold. As another example, if multiple person objects are recognized from the image, it may be determined whether the size of the area corresponding to only the object with the largest size is equal to or greater than the predetermined threshold. In this case, the predetermined threshold may be related to the absolute size of the area or a value relative to the size of the entire recognized image. This configuration may prevent additional processing time (e.g., extraction of skeletal information, detection of associated user accounts, etc.) due to person objects (e.g., people in the background) included in the image that are not intended by the photographer.

[0074] The processor can detect the user account of the instant messaging application associated with the person object in the image (S430). Specifically, the processor can recognize the face area within the recognized person object and compare the recognized face area with the face information of the user account of the instant messaging application associated with the user terminal. Additionally or alternatively, the processor can recognize the face area within the recognized person object and compare the recognized face area with the face information of the user accounts that are friends with the user account of the instant messaging application associated with the user terminal. In one embodiment, in response to determining that there is no avatar information associated with the user account, the processor can search for the avatar with the highest similarity to the person object in the image. In this case, the processor can use the avatar determined to have the highest similarity to the person object as the avatar of the user account. In other embodiments, in response to determining that there is no avatar information associated with the user account, the processor can refrain from replacing the person object in the image with an avatar.

[0075] In one embodiment, the face information of the user account associated with the user terminal can be generated based on the image included in the profile information of the user account associated with the user terminal. Additionally or alternatively, the face information of the user account associated with the user terminal can be generated based on the video call image received from the user terminal. Additionally or alternatively, the face information of the user account associated with the user terminal can be generated based on the images, videos, etc. transferred by the corresponding user account via the instant messaging application.

[0076] The processor can generate a background image by removing a person object from an image received from a user terminal (or an image sensor of the user terminal) (S440). In one embodiment, the processor can change the pixel values within the region corresponding to the person object based on the pixel values within the region other than the region corresponding to the person object in the image. For example, the processor can utilize an image conversion model based on GAN to generate a fake image in which the person object in the image is adaptively removed.

[0077] Thereafter, the processor can generate a composite image in which the person object is replaced with an avatar based on avatar information and skeleton information related to the user account (S450). In one embodiment, the processor can convert the skeleton information of the person object into avatar skeleton information based on the avatar information related to the user account. In this case, the processor can generate an avatar image based on the avatar skeleton information and the avatar information, and by inserting the generated avatar image into the background image, a composite image in which the person object is replaced with an avatar can be generated. With such a configuration, since the skeleton information of the person object is changed to match an avatar with a different body proportion from that of an actual person, an image of an avatar taking a pose identical or similar to that of the person object can be generated.

[0078] FIG. 5 is a drawing showing an example of extracting the skeleton information of a person object in an image according to an embodiment of the present disclosure. As shown in the drawing, the image (510) can be a photograph of a specific person. In one embodiment, the processor can recognize the shape of the person included in the image (510) as a person object (512). Specifically, whether the object in the image is a person object can be determined by detecting the contour line of the object and then using the shape of the contour line.

[0079] If a human object is detected in the image 510, the processor may recognize the facial region 514 of the detected human object 512 and detect a user account (e.g., a user account for an instant messaging application) associated with the human object 512. The processor may then extract skeletal information 520 of the human object 512 recognized in the image 510. As illustrated, the skeletal information may be information indicating the face size, arm and leg lengths, and pose of the human object 512 using lines and curves. While FIG. 5 illustrates the image 510 as including one human object 512, this is not limiting. For example, if the image includes multiple human objects, the processor may extract skeletal information for each human object. Additionally, the processor may be configured to extract skeletal information only for the largest human object or only for human objects having a size equal to or greater than a preset threshold.

[0080] FIG. 6 illustrates an example of generating a background image 620 from an image 610 by removing a human object 612, according to an embodiment of the present disclosure. As illustrated, a processor may recognize a human object 612 included in the image 610 and adaptively remove a region 614 corresponding to the recognized human object 612 to generate the background image 620. In one embodiment, the processor may modify pixel values in a region 614 corresponding to the human object 612 based on pixel values in regions other than the region 614 corresponding to the human object 612 in the image 610. For example, pixel values in a region corresponding to the head of the human object 612 may be modified according to pixel values in a nearby region (e.g., the window glass of a bus). Additionally or alternatively, the processor may generate a fake image from the image 610 by adaptively removing the human object 612 using a GAN-based image transformation model, etc.

[0081] With such a configuration, the area (614) corresponding to the person object (612) is removed, and pixel values based on the surrounding content are input into the corresponding area. Therefore, a background image (620) with the person object (612) naturally removed can be generated. As shown in the figure, in the background image (620), the upper body part of the person object (612) is replaced with pixel values similar to the bus part of the background, and the lower body part is replaced with pixel values similar to the road part of the background, so that the person object (612) can be naturally removed. By using such a background image (620), even if an image of another avatar with different body proportions such as the head, arms, and legs from the person object (612) is inserted, a high-quality composite image (see 720 in FIG. 7) can be generated.

[0082] FIG. 7 is a drawing showing an example of generating a composite image (720) by converting the skeleton information (520) of a person object into avatar skeleton information (710) according to an embodiment of the present disclosure. In one embodiment, since the avatar has body proportions (e.g., head, arms, legs, etc.) different from those of an actual person, the processor can obtain avatar information (e.g., the body proportions of the avatar, avatar 3D model information, etc.) related to the detected user account, and based on the obtained avatar information, convert the skeleton information (520) of the person object into avatar skeleton information (710). As shown in the figure, the avatar skeleton information (710) can have a larger head size, shorter arms, and leg lengths compared to the skeleton information (520) of the person object, and can take the same pose as the skeleton information (520) of the person object.

[0083] After that, based on the obtained avatar information and avatar skeleton information (710), the processor can generate an avatar image (722) that poses the same or a similar pose as the human object (512 in FIG. 5). The processor can generate a composite image (720) in which the human object (512 in FIG. 5) is replaced by the avatar image (722) by inserting the generated avatar image (722) into the background image (620). In this case, the avatar image (722) can be inserted at the position where the human object (512 in FIG. 5) was located. For example, the position of the tip of the foot of the avatar image (722) can be aligned with the position of the tip of the foot of the human object (512 in FIG. 5), and the position of the tip of the head of the avatar image (722) can be aligned with the position of the tip of the head of the human object (512 in FIG. 5).

[0084] FIG. 8 is a drawing showing an example in which an information processing system (200) transfers and receives information to and from a user terminal (220) according to an embodiment of the present disclosure. As shown in the drawing, the information processing system (200) can receive an image (810) including a human object from the user terminal (220) via an instant messaging application. In one embodiment, in response to receiving the image (810), the information processing system (200) can detect information (820) about a first user account associated with the human object included in the image (810) (for example, a user account of the instant messaging application). In this case, the information processing system (200) can transfer the detected information (820) about the first user account to the user terminal (220) via the instant messaging application.

[0085] The user terminal (220) can provide the user with the received information (820) regarding the first user account via a display. When the information processing system (200) misrecognizes a person object in the image (810), the user terminal (220) can, based on the user's input, transfer the information regarding the correct user account associated with the person object (e.g., the information (830) regarding the second user account) to the information processing system (200) via an instant messaging application.

[0086] Based on the received information (830) regarding the second user account, the information processing system (200) can obtain the avatar information (840) associated with the second user account and transfer the obtained avatar information (840) to the user terminal (220) via an instant messaging application. Additionally, the information processing system (200) can extract the skeletal information (850) of the person object included in the received image (810) and transfer the extracted skeletal information (850) to the user terminal (220) via an instant messaging application. Furthermore, the information processing system (200) can generate a background image (860) in which the person object in the received image (810) is adaptively removed and transfer the generated background image (860) to the user terminal (220) via an instant messaging application. The user terminal (220) can generate a composite image in which the person object is replaced with an avatar based on the avatar information (840), the skeletal information (850), and the background image (860) received from the information processing system (200).

[0087] In one embodiment, by receiving feedback on a misrecognized person object from a user and retraining a face recognition model based on the corresponding information, the accuracy of face recognition can be improved. FIG. 8 illustrates, but is not limited to, an information processing system (200) transferring information (820) for a first user account to a user terminal (220) and receiving information (830) for a second user account. For example, in response to determining that a person object is in a side or back pose of a person, a message requesting information for a user account associated with the person object can be transferred to the user terminal (220) via an instant messaging application. In this case, the information processing system (200) can determine that it is in a side or back pose of a person when the face is rotated by a predetermined angle (e.g., 60 degrees) or more from the front. For this purpose, the information processing system (200) can calculate the rotation angle of the face by analyzing the positions, sizes, etc. of eyes, nose, mouth, etc. within the face region. Alternatively, in response to determining that a person object is in a side or back pose of a person, the information processing system (200) can refrain from replacing the person object with an avatar.

[0088] Additionally or alternatively, when no user account associated with a person object is detected, the information processing system (200) can transfer a message requesting information for a user account associated with the person object to the user terminal (220) via an instant messaging application. In this case, the information processing system (200) can receive a user account associated with the person object from the user terminal (220) and can replace the person object with an avatar even when it is difficult to detect the associated user account. With such a configuration, the face recognition model can be retrained using user input information to improve the accuracy of face recognition.

[0089] FIG. 9 is a flowchart illustrating an example of a synthetic image generation method (900) for replacing a human object with an avatar based on an image capture angle according to an embodiment of the present disclosure. In one embodiment, the synthetic image generation method (900) can be initiated by a processor receiving an image (S910). Thereafter, the processor can adaptively remove an area corresponding to a human object included in the image to generate a background image (S920). Also, the processor can detect a user account of an instant messaging application related to the human object included in the image (S930). Thereafter, the processor can obtain avatar information (e.g., body proportions of the avatar, 3D model information of the avatar, etc.) related to the detected user account (S940).

[0090] Additionally, the processor can extract skeletal information of the human object included in the image (S950). In addition to this, the processor can extract shooting angle information from the image (S960). In one embodiment, the shooting angle information can be included in the image as metadata or the like. In other embodiments, the processor can analyze the image to estimate the shooting angle information. In this case, a machine learning model that has learned a large number of pairs of images and shooting angle information can be used.

[0091] Thereafter, the processor can generate a synthetic image in which the human object in the received image is replaced with an avatar based on the background image, avatar information, skeletal information, and shooting angle information (S970). With such a configuration, an avatar image reflecting perspective, distortion, size differences, etc. due to differences in shooting angles can be generated. Therefore, an avatar image more similar to the pose of the photographed human object can be generated. For example, the photographer can emphasize a specific body part (e.g., head, legs, arms, etc.) of the human object according to the shooting angle and generate an avatar image that reflects such an intention.

[0092] FIG. 10 is a drawing showing an example of replacing a human object with an avatar reflecting the shooting viewpoint of a camera according to an embodiment of the present disclosure. As shown in the drawing, a first user (1010) can photograph a second user (1030) using a user terminal (1020). For example, the first user (1010) can photograph the second user (1030) by tilting the user terminal (1020) at an angle of 50 degrees. In one embodiment, the processor can generate an avatar image (1050) by additionally considering the shooting angle information at the time of photographing. For example, the shooting angle information can be included as metadata in the image.

[0093] Specifically, the processor can extract the skeleton information of the human object and detect the user account associated with the human object. In addition, the processor can generate a background image from which the human object included in the photographed image has been removed. Then, the processor can generate a composite image (1040) in which the human object is replaced with an avatar image (1050) based on the background image, the skeleton information, the avatar information associated with the user account, and the shooting angle information. Therefore, the avatar image (1050) not only takes the same pose as the photographed human object, but can also identically reflect the perspective, distortion, size difference, etc. due to the shooting viewpoint (i.e., shooting angle) of the camera.

[0094] 11 is a diagram illustrating an example in which avatars are rendered according to the order of the users when three users (1130, 1140, 1150) are photographed, according to one embodiment of the present disclosure. As illustrated, a first user (1110) can photograph a second user (1130), a third user (1140), and a fourth user (1150) using a user terminal (1120). In this case, a processor can determine the order of the second user (1130), the third user (1140), and the fourth user (1150) and generate a composite image (1160) including the second user's avatar (1170), the third user's avatar (1180), and the fourth user's avatar (1190) based on the determined order information.

[0095] Specifically, the processor may extract skeletal information of the person objects and detect a user account associated with each person object. Additionally, the processor may determine a depth difference between person objects in the image. For example, the processor may compare the positions of the feet of the person objects, compare the sizes of the faces of the person objects, or determine the depth difference between the person objects using depth information (e.g., a depth image, a depth map, etc.) included in the image. The processor may also generate a background image from which the person objects included in the captured image have been removed.

[0096] The processor can then generate a composite image 1160 in which the three person objects are replaced with avatars 1170, 1180, and 1190 based on the background image, skeletal information of each person object, avatar information associated with each user account, and depth difference information. As shown, the third user's avatar 1180 can be displayed on the fourth user's avatar 1190, and the second user's avatar 1170 can be displayed on the third user's avatar 1180. This configuration allows the generation of a composite image 1160 that reflects the front and back positions of the people.

[0097] Additionally or alternatively, if it is determined that the distance between users included in the image is within a predetermined distance or that the shapes of the users (i.e., human objects) overlap by a predetermined area or more, the processor may generate a reduced avatar image. This configuration can prevent a rear avatar from being excessively obscured by a front avatar.

[0098] The above-described avatar service providing method can also be implemented as computer-readable code on a computer-readable recording medium. Computer-readable recording media include all types of storage devices in which data readable by a computer system is stored. Examples of computer-readable recording media include ROM, RAM, CD-ROM, magnetic tape, floppy disk, and optical data storage device. Furthermore, the computer-readable recording media can be distributed among computer systems connected to a network, and the computer-readable code can be stored and executed in a distributed manner. Furthermore, functional programs, codes, and code segments for implementing the above-described embodiments can be easily construed by programmers skilled in the art to which the present invention pertains.

[0099] The methods, operations, or techniques of the present disclosure can also be implemented by various means. For example, these techniques can also be implemented by hardware, firmware, software, or combinations thereof. Those skilled in the art will understand that the various exemplary logical blocks, modules, circuits, and algorithm steps described in connection with the disclosure of the present application can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate such interchangeability between hardware and software, various exemplary components, blocks, modules, circuits, and steps have been generally described above from a functional perspective. Whether such functions are implemented as hardware or as software will depend on the specific application and the design requirements imposed on the overall system. Those skilled in the art can implement the functions described in various ways for each specific application, but such implementation should not be construed as departing from the scope of the present disclosure.

[0100] In the case of a hardware implementation, the processing unit utilized to perform the technique can be implemented from one or more ASICs, DSPs, digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, electronic devices, other electronic units designed to perform the functions described in the present disclosure, computers, or combinations thereof.

[0101] Accordingly, the various illustrative logical blocks, modules, and circuits described in connection with the present disclosure can be implemented or performed in any combination by a general-purpose processor, DSP, ASIC, FPGA, or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination of devices designed to perform the functions described herein. A general-purpose processor can be a microprocessor, but in the alternative, the processor can be any conventional processor, controller, microcontroller, or state machine. The processor can also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other configuration.

[0102] In a firmware and / or software implementation, the techniques can be implemented with instructions stored on a computer-readable medium such as a random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), PROM (programmable read-only memory), EPROM (erasable programmable read-only memory), EEPROM (electrically erasable PROM), flash memory, compact disc (CD), magnetic or optical data storage device, etc. The instructions can be executed by one or more processors and can cause the processors to perform particular aspects of the functions described in the present disclosure.

[0103] If implemented in software, the techniques may be stored on or transmitted over as one or more instructions or code on a computer-readable medium. Computer-readable media includes any medium that facilitates transfer of a computer program from one place to another, including both computer storage media and communication media. Storage media may also be any available medium accessible by a computer. By way of non-limiting example, these computer-readable media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or different magnetic storage devices, or any other medium accessible by a computer that can be used to transport or store desired program code in the form of instructions or data structures. Also, any connection may be properly referred to as a computer-readable medium.

[0104] For example, if the software is transferred from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted wire, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted wire, digital subscriber line, or wireless technologies such as infrared, radio, and microwave are included within the definition of medium. As used herein, disk and disc include CDs, laser discs, optical discs, digital versatile discs (DVDs), floppy disks, and Blu-ray discs, where disks typically reproduce data magnetically, while discs reproduce data optically using lasers. Combinations of the above should also be included within the scope of computer-readable media.

[0105] The software module can also reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, removable disk, CD-ROM, or any other known form of storage medium. An exemplary storage medium can be connected to the processor so that the processor can read information from or record information to the storage medium. Alternatively, the storage medium can be integrated into the processor. The processor and the storage medium can also be present in an ASIC. The ASIC can also be present in a user terminal. Alternatively, the processor and the storage medium can be present as separate components in a user terminal.

[0106] While the embodiments described above have been described as utilizing aspects of the presently disclosed subject matter in one or more stand-alone computer systems, the present disclosure is not so limited and can also be implemented in conjunction with any computing environment such as a network or distributed computing environment. Further, aspects of the subject matter of the present disclosure can also be implemented in multiple processing chips or devices, and storage can be similarly affected across multiple devices. These devices can also include PCs, network servers, and handheld devices.

[0107] In this specification, although the present disclosure has been described in connection with some embodiments, various modifications and changes can be made without departing from the scope of the present disclosure that can be understood by those skilled in the art to which the invention of the present disclosure pertains. Also, such modifications and changes should be considered to fall within the scope of the appended claims of this specification.

Description of Reference Numerals

[0108] 110: First User 120: User Terminal 130: Second User 140: Composite Image 150: Avatar Image

Claims

1. In a method for providing an avatar service performed by at least one processor, receiving, via an instant messaging application, an image including a first human object from a user terminal; extracting skeletal information of the first human object; detecting a first user account of the instant messaging application associated with the first human object; generating a composite image in which the first human object is replaced by an avatar based on first avatar information associated with the first user account and the skeletal information of the first human object; and removing the first human object from the image to generate a background image. A method for providing an avatar service.

2. In a method for providing an avatar service performed by at least one processor, receiving, via an instant messaging application, an image including a first human object from a user terminal; extracting skeletal information of the first human object; detecting a first user account of the instant messaging application associated with the first human object; removing the first human object from the image to generate a background image; and transferring the first avatar information associated with the first user account, the skeletal information of the first human object, and the background image to the user terminal. A method for providing an avatar service.

3. The step of removing the first human object from the image to generate a background image includes changing pixel values within a region corresponding to the first human object based on pixel values within a region other than the region corresponding to the first human object in the image. The method for providing an avatar service according to claim 1 or 2.

4. The step of detecting a first user account of the instant messaging application associated with the first human object The method for providing an avatar service according to any one of claims 1 to 3, comprising the step of comparing the face region in the first human object with the face information of the user account of the instant messaging application associated with the user terminal.

5. The face information of the user account associated with the user terminal is generated based on at least one of at least one image included in the profile information of the user account associated with the user terminal or at least one video call image received from the user terminal. The method for providing an avatar service according to claim 4.

6. In a method for providing an avatar service performed by at least one processor, Receiving an image including a first human object from a user terminal via an instant messaging application; Extracting the skeleton information of the first human object; Detecting a first user account of the instant messaging application associated with the first human object; Removing the first human object from the image to generate a background image, The step of detecting a first user account of the instant messaging application associated with the first human object includes: A method for providing an avatar service, comprising the step of comparing the face region in the first human object with the face information of the user accounts that are friends with the user account of the instant messaging application associated with the user terminal.

7. The step of generating the composite image includes: Converting the skeleton information of the first human object into avatar skeleton information based on the first avatar information; Generating an avatar image based on the avatar skeleton information and the first avatar information; Inserting the avatar image into the background image. The method for providing an avatar service according to claim 1.

8. In a method for providing an avatar service performed by at least one processor, Receiving an image including a first human object from a user terminal via an instant messaging application; Extracting the skeleton information of the first human object; Detecting a first user account of the instant messaging application associated with the first person object; In response to determining that there is no avatar information associated with the first user account, searching for an avatar with the highest similarity to the first person object; Generating a background image by removing the first person object from the image, the avatar service providing method comprising:

9. An avatar service providing method performed by at least one processor, comprising: Receiving an image including a first person object from a user terminal via an instant messaging application; Extracting skeleton information of the first person object; Detecting a first user account of the instant messaging application associated with the first person object; Generating a background image by removing the first person object from the image, wherein the image includes shooting angle information; the avatar service providing method further comprising: Generating a composite image in which the first person object is replaced with an avatar based on the first avatar information associated with the first user account, the skeleton information of the first person object, and the shooting angle information. An avatar service providing method.

10. An avatar service providing method performed by at least one processor, comprising: Receiving an image including a first person object from a user terminal via an instant messaging application; Extracting skeleton information of the first person object; Detecting a first user account of the instant messaging application associated with the first person object; Estimating the shooting angle information of the image; Generating a composite image in which the first person object is replaced with an avatar based on the first avatar information associated with the first user account, the skeleton information of the first person object, and the shooting angle information; Generating a background image by removing the first person object from the image, the avatar service providing method comprising:

11. Claim 11 The image further includes a second person object, The avatar service providing method is, extracting the skeletal information of the second person object; detecting a second user account of the instant messaging application associated with the second person object; further including determining a difference in depth between the first person object and the second person object in the image; The step of removing the first person object from the image to generate a background image, includes the step of removing the second person object from the image to generate the background image; The step of generating the composite image, The method for providing an avatar service according to claim 1, comprising generating the composite image based on the first avatar information, second avatar information associated with the second user account, skeletal information of the first person object, skeletal information of the second person object, and the depth difference.

12. The step of determining a difference in depth between the first person object and the second person object in the image, comparing the position of the feet of the first person object with the position of the feet of the second person object, comparing the size of the face of the first person object with the size of the face of the second person object, or The method for providing an avatar service according to claim 11, including at least one of comparing the depth information of the first person object and the depth information of the second person object based on the depth information included in the image.

13. In a method for providing an avatar service performed by at least one processor, receiving, from a user terminal via an instant messaging application, an image including a first person object; extracting the skeletal information of the first person object; detecting a first user account of the instant messaging application associated with the first person object; and removing the first person object from the image to generate a background image, wherein the step of detecting a first user account of the instant messaging application associated with the first person object, In response to determining that the first human object is at least one of a lying or a standing position, a step of transferring, via the instant messaging application, a message requesting information about a user account associated with the first human object to the user terminal, an avatar service providing method.

14. In an avatar service providing method performed by at least one processor, a step of receiving, via an instant messaging application, an image including a first human object from a user terminal; a step of extracting skeletal information of the first human object; a step of detecting a first user account of the instant messaging application associated with the first human object; a step of removing the first human object from the image to generate a background image, including: The step of detecting a first user account of the instant messaging application associated with the first human object includes: a step of transferring the first user account to the user terminal via the instant messaging application; a step of receiving, via the instant messaging application, a user account different from the first user account from the user terminal, an avatar service providing method.

15. The step of extracting skeletal information of the first human object includes: a step of determining whether a size of a region corresponding to the first human object is equal to or greater than a preset threshold value, the avatar service providing method according to any one of Claims 1 to 14.

16. A computer program for causing a computer to execute the avatar service providing method according to any one of Claims 1 to 15.

17. An avatar service providing system, including: a communication module configured to receive, via an instant messaging application, an image including a human object from a user terminal; a memory; at least one processor coupled to the memory and configured to execute at least one computer-readable program included in the memory, wherein the at least one program is Extract the skeletal information of the human object, detect the user account of the instant messaging application related to the human object, generate a composite image in which the human object is replaced by an avatar based on the avatar information related to the user account and the skeletal information of the human object, An avatar service providing system including a command for removing the human object from the image to generate a background image.

18. An avatar service providing system, a communication module configured to receive an image including a human object from a user terminal via an instant messaging application, a memory, at least one processor coupled to the memory and configured to execute at least one computer-readable program included in the memory, wherein the at least one program, extracts the skeletal information of the human object, detects the user account of the instant messaging application related to the human object, removes the human object from the image to generate a background image, An avatar service providing system including a command for transferring the avatar information related to the user account, the skeletal information of the human object, and the background image to the user terminal.

19. An avatar service providing system, a communication module configured to receive an image including a human object from a user terminal via an instant messaging application, a memory, at least one processor coupled to the memory and configured to execute at least one computer-readable program included in the memory, wherein the at least one program, extracts the skeletal information of the human object, detects the user account of the instant messaging application related to the human object, includes a command for removing the human object from the image to generate a background image, Detecting the user account of the instant messaging application related to the human object is, An avatar service providing system, comprising comparing a face region within the human object with face information of user accounts that are in a friendship relationship with the user account of the instant messaging application associated with the user terminal. **Claim 20**: An avatar service providing system, A communication module configured to receive an image including a human object from a user terminal via an instant messaging application, A memory, At least one processor coupled to the memory and configured to execute at least one computer-readable program included in the memory, The at least one program includes: Extracting skeleton information of the human object, Detecting a user account of the instant messaging application associated with the human object, In response to determining that there is no avatar information associated with the user account, searching for an avatar with the highest similarity to the human object, An avatar service providing system including instruction words for removing the human object from the image to generate a background image. **Claim 21**: An avatar service providing system, A communication module configured to receive an image including a human object from a user terminal via an instant messaging application, A memory, At least one processor coupled to the memory and configured to execute at least one computer-readable program included in the memory, The at least one program includes: Extracting skeleton information of the human object, Detecting a user account of the instant messaging application associated with the human object, Including instruction words for removing the human object from the image to generate a background image, The image includes shooting angle information, The at least one program includes: An avatar service providing system further including instruction words for generating a composite image in which the human object is replaced by an avatar based on the avatar information associated with the user account, the skeleton information of the human object, and the shooting angle information. **Claim 22**: An avatar service providing system, A communication module configured to receive an image including a human object from a user terminal via an instant messaging application; A memory; At least one processor coupled to the memory and configured to execute at least one computer-readable program included in the memory, The at least one program includes: Extracting the skeletal information of the human object; Detecting the user account of the instant messaging application associated with the human object; Estimating the shooting angle information of the image; Generating a composite image in which the human object is replaced by an avatar based on the avatar information associated with the user account, the skeletal information of the human object, and the shooting angle information; An avatar service providing system including instructions for removing the human object from the image to generate a background image.

23. An avatar service providing system, A communication module configured to receive an image including a human object from a user terminal via an instant messaging application; A memory; At least one processor coupled to the memory and configured to execute at least one computer-readable program included in the memory, The at least one program includes: Extracting the skeletal information of the human object; Detecting the user account of the instant messaging application associated with the human object; Including instructions for removing the human object from the image to generate a background image, Detecting the user account of the instant messaging application associated with the human object includes: In response to determining that the human object is at least one of a lateral or rear view of a person, transferring a message requesting information about the user account associated with the human object to the user terminal via the instant messaging application. An avatar service providing system.

24. An avatar service providing system, A communication module configured to receive an image including a person object from a user terminal via an instant messaging application; A memory; At least one processor coupled to the memory and configured to execute at least one computer-readable program included in the memory, The at least one program includes: Extracting skeletal information of the person object; Detecting a user account of the instant messaging application associated with the person object; Including instructions for removing the person object from the image to generate a background image; Detecting the user account of the instant messaging application associated with the person object includes: Transferring the user account to the user terminal via the instant messaging application; An avatar service providing system including receiving, via the instant messaging application, a user account different from the user account from the user terminal.

25. In an avatar service providing method performed by at least one processor, Receiving an image including a person object; Transferring the image to an external device via an instant messaging application; Receiving the skeletal information of the person object from the external device; Receiving avatar information associated with the person object from the external device; Receiving a background image with the person object removed from the image from the external device; Generating a composite image in which the person object is replaced by an avatar based on the skeletal information, the avatar information, and the background image. An avatar service providing method.

26. In an avatar service providing method performed by at least one processor, Receiving an image including a person object; Transferring at least a part of the image to an external device via an instant messaging application; Receiving avatar information associated with the person object from the external device; Extracting the skeletal information of the person object; A step of removing the human object from the image to generate a background image; An avatar service providing method, including a step of generating a composite image in which the human object is replaced with an avatar based on the received avatar information, the skeleton information, and the background image.

Images