Computing systems, methods, storage media, server devices, and computer programs

Abstract

The computing system provides a social media platform. The computing system comprises one or more processors configured to receive a base image containing a human face and an image mask defining areas where inpainting occurs and areas where inpainting does not occur, by executing instructions stored in associated memory. The areas where inpainting does not occur include at least the eye area. One or more processors are configured to accept user text prompts and to use the base image, image mask, and user text prompts as input to an artificial intelligence (AI) model to generate a face decoration texture.

Description

Technical Field

[0003] , ,

[0001] Cross - reference to Related Applications This application claims priority to U.S. Provisional Patent Application No. 63 / 505,346, filed on May 31, 2023, and U.S. Patent Application No. 18 / 354,546, filed on July 18, 2023, the entire contents of which are incorporated herein by reference for all purposes.

Background Art

[0002] Many social media platforms provide tools for users to add effects to images and videos before publishing the content online. Some of these effects, such as filters, stickers, and textures, which are designed to make objects or materials appear to exist in images and videos when they do not actually exist, or to modify or extend real - world objects, are applied on human faces. These effects are usually provided in an effect library, and in some social media platforms, users can create new effects by themselves. Since the creation is usually done manually, for example, in image - editing software, it is also limited to advanced users. At the same time, artificial intelligence (AI) is becoming increasingly popular as a tool for generating images without humans having to create them from scratch manually. So far, attempts to use AI - generated images for creating new effects on social media have required further manual adjustments to complete the effects, thus limiting the practicality of AI in this area and preventing the creation of effects by the general public.

Summary of the Invention

Means for Solving the Problems

[0003] A computing system for providing a social media platform is provided herein. In one example, the computing system comprises one or more processors configured to receive a base image including a human face and an image mask defining areas where inpainting occurs and areas where inpainting does not occur, by executing instructions stored in associated memory. The areas where inpainting does not occur include at least the eye area. The one or more processors are configured to accept a user text prompt and to generate a face decoration texture in an artificial intelligence (AI) model using the base image, the image mask, and the user text prompt as input.

[0004] This summary is provided to present a simplified excerpt of the concept, which will be further described in the embodiments for carrying out the invention described below. This summary is not intended to identify the main or basic features of the claimed subject matter, nor to limit the scope of the claimed subject matter. Furthermore, the claimed subject matter is not limited to an implementation that solves any or all of the defects described in any part of this disclosure. [Brief explanation of the drawing]

[0005] [Figure 1] This is a schematic diagram of a computing system, including server equipment, that provides a social media platform.

[0006] [Figure 2] This figure shows an exemplary artificial intelligence (AI) model used by the computing system shown in Figure 1 for generating face decoration textures.

[0007] [Figure 3] Figures 3A and 3B show exemplary base images used by the computing system in Figure 1.

[0008] [Figure 4] Figures 4A to 4D show exemplary image masks used by the computing system in Figure 1.

[0009] [Figure 5] This figure shows an example graphical user interface (GUI) of a social media platform, displaying a prompt input screen.

[0010] [Figure 6] Figure 5 shows another example prompt input screen of the GUI.

[0011] [Figure 7] Figure 5 shows an example of the GUI output selection screen.

[0012] [Figure 8A] Figure 5 shows an example of a video editing screen with a GUI.

[0013] [Figure 8B] Figure 8A shows an example video being edited within the video editing screen, where an "aged woman" face decoration texture has been applied to a woman's face. [Figure 8C] Figure 8A shows an example video being edited within the video editing screen, where an "aged woman" face decoration texture has been applied to a woman's face.

[0014] [Figure 9] Figures 9A and 9B show an exemplary video being edited within the video editing screen of Figure 8A, where the "aged woman" face decoration texture is blended for skin tone matching and applied to the woman's face.

[0015] [Figure 10]FIG. 5 is a diagram showing an exemplary blend menu of the GUI.

[0016] [Figure 11] FIGS. 11A-11C are diagrams showing another exemplary video being edited within the video editing screen of FIG. 8A, in which a makeup face decoration texture is applied to a female face.

[0017] [Figure 12] FIG. 5 is a diagram showing an exemplary effect menu that lists trendy effects within the GUI.

[0018] [Figure 13] FIG. 18 is a flowchart of a method for a social media platform.

[0019] [Figure 14] FIG. 24 is a schematic diagram showing an exemplary computing environment capable of implementing the computing system of FIG. 1.

BEST MODE FOR CARRYING OUT THE INVENTION

[0020] To solve the above problem, Figure 1 shows a computing system 100 including a server device 10 that provides a social media platform 12. The server device 10 comprises one or more processors 14, which execute instructions stored in associated memory 16 to realize various functions of the server device 10. The instructions may include, for example, a face decoration texture generation module 18 and an application server program 20. It will be understood that the server device 10 may include multiple different servers working together to provide the social media platform 12, or it may be a single server. The server device 10 may further include an effect data store 22 for storing a library of face decoration textures for use by users of the social media platform 12, and a video data store 24 for storing published video content for viewing by users.

[0021] One or more processors 14 may be configured to send commands to the client device 26 to display the graphical user interface (GUI) 28 of the social media platform 12 on the client device 26. The server device 10 and the client device 24 may communicate with each other via the network 30 and one or more handlers 32 of the application server program 22. The client device 26 may be a smartphone, tablet, personal computer, etc., including one or more processors 34 configured to execute the client program 36 and display the GUI 28 on the display 30, memory 32 for storing commands, and one or more input devices 34 for receiving user input. The input devices 34 may include, for example, a touchscreen, keyboard, microphone, camera, accelerometer, etc. It will be understood that the client program 36 may be a dedicated application for accessing the social media platform 12, or it may be a general program, such as an internet browser for accessing content from various server devices, including the social media platform 12, from the server device 10. Furthermore, it will be understood that in some implementations, the face decoration texture generation module 18 may be executed locally by one or more processors 34 of the client device 26.

[0022] In short, the client device 24 may send a generation request 38 to the server device 10 requesting the generation of a new effect. One or more processors 14 may be configured to accept the generation request 38, which includes a base image selection 40 and a user text prompt 42. The artificial intelligence (AI) model 44 may receive, as input, a base image 46 containing a human face, an image mask 48 defining areas where inpainting occurs and areas where inpainting does not occur (see the example in Figures 4A to 4D described below), and a user text prompt 42. The AI ​​model 44 may be configured to receive the base image 46 from the user of the client device 26, but more simply, it will be understood that receiving the base image 46 may include receiving a selection 40 for one of several base images 46 (see Figures 3A to 3B for two examples). Alternatively, the AI ​​model 44 may be configured to retrieve a single stored base image 46 from memory 16. By providing the face decoration texture generation module 18 with multiple base images 46, users can test their creations on people with significantly different appearances to ensure that the created effects are suitable for a wide range of users. In some implementations, the face decoration texture generation module 18 may include multiple models, including other AI models 44A, while in other implementations, only one AI model 44 may be used instead.

[0023] The face decoration texture generation module 18 may be configured in the AI ​​model 44 to generate a face decoration texture 50 using the base image 46, image mask 48, and user text prompt 42 as input. As will be described in more detail below, the areas where inpainting does not occur include at least the eye area, which contributes to both the AI ​​model 44 centering the generated face decoration texture 50 in the correct position on the human face in the base image, and the face decoration texture generation module 18 centering the generated face decoration texture 50 in the correct position on the human face in the acquired image or video, using at least the eyes as anchor points. Finally, the server device 10 may be configured to store the face decoration texture 50 in the effect data store 22 and / or to send the face decoration texture 50 to the client device 26.

[0024] Figure 2 shows an example of an AI model 44 used by the computing system 100 of Figure 1 for generating face decoration textures. The shown AI model 44 is merely an example, and any suitable generative AI model may be used. In the illustrated example, the AI ​​model 44 is a trained machine learning model, more specifically, a diffusion model. Examples of known diffusion models may include stable diffusion, realistic vision, etc., and a suitable diffusion model may include modified versions of these known models. In this example, an image encoder 52 is provided, comprising a pre-trained layer 52A such as a pre-trained contrastive language-image pre-training (CLIP) vision transformer (ViT), a fine-tuning layer 52B trained to extract visual features from input images such as a base image 46 and an image mask 48, and a fully connected layer 52C configured to generate a set of embeddings 54 based on the visual features of faces in the base image extracted by at least the fine-tuning layer 52B. The image mask 48 may be treated as an alpha channel indicating which parts of the output should be opaque (included) and which parts should be transparent (excluded). For example, the base image 46 may first be masked based on the blocked and avoided areas of the image mask 48, and the resulting masked base image may be used as input for generation as described below. In some implementations, a set of embeddings 54 may be associated with the user identifier 56 of the user of the client device 26.

[0025] The AI ​​model 44 may be configured to accept a user text prompt 42 in which the user describes what effect the AI ​​model 44 should create. The user text prompt 42 and a set of embeddings 54 may be provided as input to a text encoder 58, which may generate an input feature vector 60 based on at least the user text prompt 42 and the set of embeddings 54. This input feature vector 60 may be sent to a diffusion module 62 of the AI ​​model 44, which is configured to generate a composite image as a face decoration texture 50 based on at least the input feature vector 60.

[0026] Figures 4A to 4D show examples of image masks 48 used by the computing system 100 in Figure 1. In Figure 4A, the generated face decoration texture 50 is a mask, and the first image mask 48A includes areas where inpainting occurs (white) and areas where inpainting does not occur (black). Here, the areas where inpainting does not occur include the eye areas 64 (specifically, the areas of the two eyes 64) as well as the mouth area 66. A common shape is, for example, a face mask for a Halloween costume, so the areas where inpainting occurs are limited to this face mask shape and do not include the area 68 surrounding the face. By using the first image mask 48A, a full-face mask with the mouth and eyes cut out can be output as the face decoration texture 50 so that the eyes and mouth of the person in the image are visible. In some implementations, the face decoration texture 50 may be makeup, as in the case of using the exemplary image masks 48B to 48D shown in Figures 4B to 4D. In Figure 4B, the second image mask 48B includes the mouth region 66, which is part of the area where inpainting occurs (white), and the area where inpainting does not occur (black) further includes the area 70 surrounding the mouth region 66. This is in particular the opposite of the first image mask 48, in which the mouth region 66 is included in the area where inpainting does not occur (black). This is because face masks typically show the wearer's mouth, whereas makeup typically includes lipstick or lip gloss on the wearer's mouth. Therefore, the second image mask 48 may be lip-shaped, and the area 70 surrounding the lips may be excluded from the area where inpainting occurs.

[0027] Figures 4C to 4D all include areas 72 around the eye area 64 in the region where inpainting occurs. For example, the third image mask 48C in Figure 4 has an area (white) where inpainting occurs that irregularly surrounds the eye area 64, providing a palette primarily of eyeshadow above the wearer's eyes and additional under-eye makeup that more lightly surrounds the lower part of the eye area 64. It will be understood that areas 72 of different shapes around the eye area 64, for example, one wider area surrounding two eye areas 66, may be used. The fourth image mask 48D in Figure 4D simultaneously provides only the eyelash area 74 that radiates outward from the eye area 64 and surrounds the eye area 64 as the area where inpainting occurs (white), with the rest of the image mask being black. The image mask 48 used by the AI ​​model 44 may be appropriately selected based on the expected output of the face decoration texture generation module 18, i.e., whether the user has requested the generation of a mask or makeup. The social media platform 12 may provide both functions through separate channels, determine which was requested based on the context of the user text prompt, or exclusively provide only one or only the other. Furthermore, face decoration textures other than makeup and masks may be generated. For makeup, any suitable combination of image masks 48B to 48D may be used as the image mask 48 input to the AI ​​model 44. For example, a user requesting "glitter pink lip gloss" may, in some cases, receive a generated face decoration texture 50 that covers only the mouth area 66, while a user requesting "full-glam blue makeup" may receive a face decoration texture 50 that covers both the mouth area 66 and the area 72 surrounding the eye area 64.

[0028] Figures 5A to 12 show various exemplary screens and videos (or still images) displayed by the GUI 28 in connection with the generation of face decoration textures 50. In Figures 5 and 6, the GUI 28 displays a prompt input screen 76. The example shown in Figure 5 may be adapted for a desktop version of the client program 36, and the example shown in Figure 6 may be adapted for a mobile version. The desktop version may be for more experienced users, giving them more options and control over effect creation, while the mobile version may be simplified to allow less experienced users to create effects. In Figure 5, the GUI 28 may display a prompt input box 78 configured to accept a user text prompt 42. Instructions 80 may explain how to use the face decoration effect generation function. A base image 46 selected by the user may be displayed for reference. The generation selector 82 may be selectable to send the input to the AI ​​model 44 to start generation. In contrast, in Figure 6, one or more processors 14, 34 may be configured to present a GUI 28 to a user of a client device 26, which in this case may be a mobile computing device. Here, the GUI 28 may ultimately display a face decoration texture 50 on a human face (see, for example, Figures 8B-8C), but may not display the base image 46. In both versions, the GUI 28 may not display an image mask 48 to the user. By reducing the display of extraneous inputs that the user may not be familiar with, processing can be simplified, and the GUI 28 can be made less confusing for the user. Furthermore, the mobile version may include one or more suggestion prompts 84, which may be accompanied by images or videos of the corresponding face decoration textures. Depending on the user's selection of one of the suggestion prompts 84, the suggestion prompt 84 may be added to the prompt input box 78 for the user, and the user may freely modify or add to the suggestion prompts 84 before completing the user text prompt 42.

[0029] Figure 7 shows an exemplary output selection screen 86 of the GUI 28 in Figure 5. Here, the output of the AI ​​model 44 includes several (four in this example) face decoration textures 50. The GUI 28 may include a check selector 88 for each of the face decoration textures 50 that the user wishes to keep. Each check selector 88 may be hidden, for example, until the user's cursor hovers over the respective face decoration texture 50. The import selector 90 may be operable to download all selected face decoration textures 50 to the client device 26, or, if none are selected, to download all face decoration textures 50 to the client device 26. To the right of the face decoration textures 50 is an options pane 92. A base image menu 92A may be included to allow the user to select which base image is displayed below the face decoration texture 50. This may allow the user to test whether the generated face decoration textures 50 are suitable for various face types, especially different skin tones. The options pane 92 may further include customizable options for the AI ​​model 44, such as a generation step 92B indicating the number of diffusion steps performed by the AI ​​model 44, and a prompt intensity 92C indicating the intensity with which the AI ​​model weights the user text prompt 42 during generation. Once the user is satisfied with the face decoration texture 50 and has downloaded one or more to the client device 26, the user can use the face decoration texture 50 for video and image editing.

[0030] Figure 8A shows an exemplary video editing screen 94 of the GUI 28 in Figure 5. Several icons 96 arranged around the video editing screen 94 may be operable to perform various editing tasks to produce the final video. For brevity, the icons 96 are shown only in Figure 8A. In this example, a woman is shown in the video. Once the face decoration texture 50 is created, one or more processors 14, 34 may be configured to automatically apply the face decoration texture 50 to the video acquired by the camera of the client device 26, or it may be selectable from a menu, for example, an effects menu screen 98 (see Figure 12) which can be opened via the effects selector 102. It will be understood that the video may be a live preview of the "viewfinder" of the scene that can be acquired by the camera before recording, live video material currently being recorded by the camera, or previously recorded and stored video material. For example, Figures 8B-8C show a face decoration texture 50 generated from a user text prompt "old woman" applied to a human face 104 in a live video feed. Typically, such 3D effects are applied as textures on a mesh, where the mesh tracks the human face as it moves in each frame. Human faces 104 in the live video feed are detected using a face detection algorithm (e.g., to find the eyes and mouth), and a 3D face model (mesh) is generated from the detected human faces 104 using a 3D reconstruction algorithm. The face decoration texture 50 is applied to the 3D face model. The position and orientation of the 3D model and the face decoration texture 50 applied to it are updated based on the changes in the position and orientation of the detected human faces 104 within each frame of the live video feed.

[0031] In some implementations, the face decoration texture generation module 18 may even be able to adjust the mesh to generate a tiger with a snout protruding from the wearer's face, rather than a human nose with 3D features, such as tiger stripes. This can be achieved, for example, through algorithmic depth estimation and corresponding adjustments to the image mask 48. Thus, whether the mesh is original or modified, the user can try out the face decoration texture 50 in real time with various poses, postures, and expressions, and can start shooting with its effects even if the face decoration texture 50 was created just seconds ago. Once the video is complete, the user may publish the video content 106 on the social media platform 12 for viewing by other users on other client devices 108. Other users may view the video content 106 and other video content 110 stored in the video data store 24 of the server device 10.

[0032] However, the face decoration texture 50 shown in Figures 8B to 8C has a significantly different skin tone from the woman to whom it is applied. As a result, users may not want to use the face texture generation function if they feel that the face decoration texture 50 was not made with them in mind. Therefore, one or more processors 14, 34 may be configured to determine the hue of the skin tone of the human face 104 at a pixel level, and then compare it with the hue of the corresponding pixel of the face decoration texture 50 that is overlaid on the human face 104. If the difference between the hue of the face decoration texture 50 and the hue of the skin tone is less than or equal to a threshold, one or more processors 14, 34 may return the pixel of the face decoration texture 50 as is. However, if the difference is greater than the threshold, one or more processors 14, 34 may multiply the hue of the face decoration texture 50 by the hue of the skin tone and return the resulting value as the pixel of the face decoration texture 50. Figures 9A and 9B show the same woman with slightly different “aged woman” face decoration textures 50 that have already been appropriately blended to match her skin tone. Therefore, even if the base image is a human base image with a significantly different skin tone than the human to which the generated face decoration texture 50 is applied, the skin tone can be blended, and the user can be provided with a tone-matched effect. Furthermore, because the blending process is based on the hue of the skin tone rather than another value such as brightness, warm tones may be blended more, while cool tones may be blended less. As a result, the blue eyeshadow will blend so well that it does not look like unnatural ink on a human face, for example, nor does it appear to soak into the skin of the human face. Furthermore, as shown in Figure 10, one or more processors 14, 34 may be configured to present the user of the client device 26 with multiple blend modes 112 for blending the face decoration texture 50 with the human face 104.Blend mode 112 is shown here in the example blend menu 114, and it will be understood that these are merely examples and other suitable blend modes may be included.

[0033] Figures 11A–11C show another exemplary video being edited within the video editing screen of Figure 8A, with a makeup face decoration texture applied to a woman's face. Figure 11A shows a video containing a different human face 104 of a different woman from the woman in Figure 8A, without the effect applied. In Figures 11B–11C, the face decoration texture 50 is applied to the human face 104, specifically to a mesh that is constantly updated to track the human face 104 throughout the video. In this example, the face decoration texture 50 includes eye makeup and lip makeup, but the area around the lips and the area immediately surrounding the eyes, as well as the eye region 64 itself (e.g., the visible part of the eyeball, inside the eyelids), are excluded from the applied effect. As can be seen from Figures 11B–11C, the makeup effect is precisely positioned, even when the woman in the video turns her head, because the AI ​​model 44 precisely positions the effect around the eyes and mouth of the base image 46 by using an image mask 48 for guidance. Therefore, it is possible to precisely position the generated face decoration texture 50 without manual human intervention during or after generation.

[0034] In some cases, one or more processors 14, 34 may further store face decoration textures 50 and be configured to make face decoration textures 50 available to other users of the social media platform 12 via other client devices 108. Face decoration textures 50 may be available by different means, such as the effect library of the effect data store 22, which may be accessible via the effect menu screen 98 opened by the effect selector 102 shown in Figure 8A. In some cases, one or more processors 14, 34 may further be configured to present multiple trending face decoration textures 50, including face decoration texture 50, to other users. Figure 12 shows an example of the effect menu screen 98 in the GUI 28 of Figure 5, which lists trending effects. Trending face decoration textures 50 may be engaged with (viewed, liked, applied, published, edited, etc.) by other users of the social media platform 12, as determined according to the algorithm. By allowing users to share and use user-created effects, the user base of social media platforms can have an enhanced user experience with increased options, including more options that encompass users of various ethnicities and facial types, which may also lead to increased user engagement.

[0035] Figure 13 shows a flowchart of Method 1300 for a social media platform according to this disclosure. Method 1300 may be implemented by the computing device 100 shown in Figure 1. In 1302, Method 1300 may include receiving a base image including a human face. In 1304, Method 1300 may include receiving an image mask that defines areas where inpainting occurs and areas where inpainting does not occur, including at least the eye area. In 1306, Method 1300 may include receiving a user text prompt. In 1308, Method 1300 may include generating a face decoration texture in an artificial intelligence (AI) model using the base image, the image mask, and the user text prompt as input. Thus, even users without graphic design skills can generate customized face decoration textures on the fly, and because an image mask that precisely specifies areas where inpainting occurs and areas where it does not occurs is used, the generated texture will be applied accurately and easily. In 1310, method 1300 may include applying a face decoration texture to a human face in a live video feed. Thus, once created, the face decoration texture can be immediately used by the user or other users in a live video feed that has constantly updating frames, and the face decoration texture can be positioned and held on the human face with high precision.

[0036] In some implementations, the AI ​​model may be a diffuse model. A diffuse model may be suitable for producing the desired effect. In 1312, method 1300 may include performing skin tone blending on a pixel-by-pixel basis by performing the following substeps: In 1314, determine the hue of the skin tone of a human face in the pixel; in 1316, compare it with the hue of the corresponding pixel of a face decoration texture overlaid on the human face; in 1318, if the difference between the hue of the face decoration texture and the hue of the skin tone is less than or equal to a threshold, return the pixel of the face decoration texture as is; in 1320, if the difference is greater than the threshold, multiply the hue of the face decoration texture by the hue of the skin tone and return the resulting value as the pixel of the face decoration texture. Thus, the hue of the face decoration texture can be adjusted so that the skin tone blends naturally into the acquired image of the human face, even if the base image has light skin and the human face to which the face decoration texture is applied has significantly dark skin.

[0037] In some implementations, the face decoration texture may be a mask, and the area where inpainting does not occur may further include the area around the mouth. Therefore, the generated face decoration texture can accurately track the eyes and mouth of a human face and appropriately represent these features through the face decoration texture. In other implementations, the face decoration texture may be makeup, and the area where inpainting occurs may include the area around the mouth and eyes, while the area where inpainting does not occur may further include the area around the mouth. Unlike the implementation of a mask, which should not cover the mouth of a human face, makeup, which typically includes a lip component, should cover the mouth of a human face but should not extend beyond the mouth. Therefore, even when the mouth area is included rather than excluded as in the mask implementation, the generated face decoration texture can accurately track the eyes and mouth of a human face and appropriately represent these features through the face decoration texture.

[0038] In some implementations, receiving a base image may include receiving a selection for one of several base images. For example, a desktop version of a client program may offer the user more customization features and options, allowing the user to select from various base images when requesting the generation of a face decoration texture. Alternatively, in 1322, method 1300 may include presenting a user of a mobile computing device with a graphical user interface (GUI) configured to display the face decoration texture on a human face without displaying the base image. Thus, in a simpler approach, the AI ​​model may automatically receive the base image along with an image mask used as input from a storage device, without requiring the user to provide further input to select or provide these images. In 1324, method 1300 may include storing the face decoration texture and making it available to other users of the social media platform. This improves the user experience across the social media platform, as the user can share their creations with other users.

[0039] In some embodiments, the methods and processes described herein may be linked to a computing system of one or more computing devices. Specifically, such methods and processes may be implemented as computer application programs or services, application programming interfaces (APIs), libraries, and / or other computer program products.

[0040] Figure 14 schematically illustrates a non-limiting embodiment of a computing system 1400 capable of implementing one or more of the methods and processes described above. The computing system 1400 is shown in a simplified form. The computing system 1400 can embody the computer device 10 shown in Figure 2, as described above. The computing system 1400 may take the form of one or more personal computers, server computers, tablet computers, home entertainment computers, network computing, game devices, mobile computing devices, mobile communication devices (e.g., smartphones) and / or other computing devices, and wearable computing devices, such as smartwatches and head-mounted augmented reality devices.

[0041] The computing system 1400 comprises a logical processor 1402, a volatile memory 1404, and a non-volatile storage device 1406. The computing system 1400 may optionally include a display subsystem 1408, an input subsystem 1410, a communication subsystem 1412, and / or other components not shown in Figure 14.

[0042] The logical processor 1402 includes one or more physical devices configured to execute instructions. For example, the logical processor may be configured to execute instructions that are part of one or more applications, programs, routines, libraries, objects, components, data structures, or other logical structures. Such instructions may be implemented to perform tasks, realize data types, convert the state of one or more components, achieve technical effects, or achieve desired results.

[0043] A logical processor may include one or more physical processors (hardware) configured to execute software instructions. Furthermore, or alternatively, a logical processor may include one or more hardware logic circuits or firmware devices configured to execute hardware-implemented logic or firmware instructions. The processor of the logical processor 1402 may be single-core or multi-core, and the instructions executed thereon may be configured for sequential, parallel, and / or distributed processing. Individual components of the logical processor may optionally be distributed across two or more separate devices located remotely and / or configured for coordinated processing. Embodiments of the logical processor may be virtualized and executed by computing devices connected to a remotely accessible network configured in a cloud computing setup. In such cases, it should be understood that these virtualized embodiments run on different physical logical processors on various different machines.

[0044] The non-volatile storage device 1406 includes one or more physical devices configured to hold instructions executable by a logical processor in order to implement the methods and processes described herein. When such methods and processes are implemented, the state of the non-volatile storage device 1406 may be transformed, for example, to hold different data.

[0045] The non-volatile storage device 1406 may include removable and / or built-in physical devices. The non-volatile storage device 1406 may include optical memory (e.g., CD, DVD, HD-DVD, Blu-ray disc, etc.), semiconductor memory (e.g., ROM, EPROM, EEPROM, flash memory, etc.), and / or magnetic memory (e.g., hard disk drive, floppy disk drive, tape drive, MRAM, etc.), or other mass storage technology. The non-volatile storage device 1406 may include non-volatile, dynamic, static, read / write, read-only, sequential access, position-addressable, file-addressable, and / or content-addressable devices. It will be understood that the non-volatile storage device 1406 is configured to retain instructions even when power to the non-volatile storage device 1406 is cut off.

[0046] The volatile memory 1404 may include a physical device that provides random access memory. The volatile memory 1404 is typically used by the logical processor 1402 to temporarily store information during the processing of software instructions. It will be understood that if power to the volatile memory 1404 is cut off, the volatile memory 1404 will typically no longer store instructions.

[0047] The embodiments of the logic processor 1402, the volatile memory 1404, and the non-volatile storage device 1406 may be integrated together in one or more hardware logic components. Such hardware logic components may include, for example, field-programmable gate arrays (FPGAs), program- and application-specific integrated circuits (PASICs / ASICs), program- and application-specific standard products (PSSPs / ASSPs), system-on-a-chips (SOCs), and complex programmable logic devices (CPLDs).

[0048] The terms “module,” “program,” and “engine” may be used to describe a form of computing system 1400 that is typically implemented in software, to execute a specific function, which involves a conversion process that specially configures the processor to perform a certain function using a portion of volatile memory. Therefore, a module, program, or engine may be instantiated using a portion of volatile memory 1404 via a logic processor 1402 that executes instructions held by a non-volatile memory 1406. It will be understood that different modules, programs, and / or engines may be instantiated from the same application, service, code block, object, library, routine, API, function, etc. Similarly, the same module, program, and / or engine may be instantiated from different applications, services, code block, object, routine, API, function, etc. The terms “module,” “program,” and “engine” may include individuals or groups such as executable files, data files, libraries, drivers, scripts, database records, etc.

[0049] The display subsystem 1408, if included, may be used to present a visual representation of the data held by the non-volatile memory 1406. This visual representation may take the form of a graphical user interface (GUI). Since the methods and processes described herein modify the data held by the non-volatile memory and transform the state of the non-volatile memory, the state of the display subsystem 1408 may also be transformed to visually represent the changes in the underlying data. The display subsystem 1408 may include one or more display devices utilizing substantially any type of technology. Such display devices may be combined with the logical processor 1402, volatile memory 1404 and / or non-volatile memory 1406 within a shared enclosure, or such display devices may be peripheral display devices.

[0050] The input subsystem 1410, if included, may include one or more user input devices, such as a keyboard, mouse, touchscreen, game controller, microphone, camera, accelerometer, gyroscope, and / or any other suitable senses, and may interact with them. The communication subsystem 1412, if included, may be configured to connect the various computing devices described herein to each other or to other devices in a communicative manner. The communication subsystem 1412 may include wired and / or wireless communication devices compatible with one or more different communication protocols. In non-limiting examples, the communication subsystem may be configured to communicate via a wireless telephone network, or via a wired or wireless local or wide area network, such as HDMI® over a wireless network connection. In some embodiments, the communication subsystem may enable the computing system 1400 to send and receive messages with other devices over a network such as the Internet.

[0051] The following paragraphs provide further explanation of the subject matter of this disclosure. One embodiment provides a computing system that provides a social media platform. The computing system includes one or more processors configured to receive a base image including a human face, receive an image mask defining areas where inpainting occurs and areas where inpainting does not occur, including at least the eye area, by executing instructions stored in associated memory, and to receive a user text prompt, and to use the base image, the image mask, and the user text prompt as input to generate a face decoration texture in an artificial intelligence (AI) model. In this embodiment, further or alternatively, the AI ​​model is a diffusion model. In this embodiment, further or alternatively, the face decoration texture is a mask, and the areas where inpainting does not occur further include the mouth area. In this embodiment, further or alternatively, the face decoration texture is makeup, and the areas where inpainting occurs include the areas around the mouth and the eye area, and the areas where inpainting does not occur further include the areas around the mouth area. In this embodiment, further or alternatively, receiving the base image includes receiving a selection for one of a plurality of base images. In this embodiment, the one or more processors are further configured to apply the face decoration texture over a human face in a live video feed. In this embodiment, the one or more processors are further configured to determine, on a pixel-by-pixel basis, the hue of the skin tone of the human face in a pixel, compare it to the hue of the corresponding pixel of the face decoration texture overlaid on the human face, return the pixel of the face decoration texture as is if the difference between the hue of the face decoration texture and the hue of the skin tone is less than or equal to a threshold, and return the pixel of the face decoration texture as is if the difference is greater than the threshold, multiply the hue of the face decoration texture by the hue of the skin tone and return the resulting value as the pixel of the face decoration texture.In this embodiment, the one or more processors are further configured to present to the user of the client device a plurality of blend modes for blending the face decoration texture with the human face. In this embodiment, the one or more processors are further configured to present to the user of the mobile computing device a graphical user interface (GUI) configured to not display the base image and to display the face decoration texture on top of the human face. In this embodiment, the one or more processors are further configured to store the face decoration texture and make the face decoration texture available to other users of the social media platform.

[0052] Another embodiment provides a method for a social media platform. The method includes receiving a base image including a human face; receiving an image mask defining areas where inpainting occurs and areas where inpainting does not occur, including at least the eye area; receiving a user text prompt; and generating a face decoration texture in an artificial intelligence (AI) model using the base image, the image mask, and the user text prompt as input. In this embodiment, further or alternatively, the AI ​​model is a diffusion model. In this embodiment, further or alternatively, the face decoration texture is a mask, and the areas where inpainting does not occur further include the mouth area. In this embodiment, further or alternatively, the face decoration texture is makeup, and the areas where inpainting occurs include the areas around the mouth and eye areas, and the areas where inpainting does not occur further include the areas around the mouth area. In this embodiment, further or alternatively, receiving the base image includes receiving a selection for one of a plurality of base images. In this embodiment, further or alternatively, the method further includes applying the face decoration texture onto a human face in a live video feed. In this embodiment, the method further includes, on a pixel-by-pixel basis, determining the hue of the skin tone of the human face in a pixel; comparing this hue to the hue of the corresponding pixel of the face decoration texture overlaid on the human face; returning the pixel of the face decoration texture as is if the difference between the hue of the face decoration texture and the hue of the skin tone is less than or equal to a threshold; and multiplying the hue of the face decoration texture by the hue of the skin tone and returning the resulting value as the pixel of the face decoration texture if the difference is greater than the threshold. In this embodiment, the method further includes, or rather, storing the face decoration texture and making the face decoration texture available to other users of the social media platform.In this embodiment, a non-temporary computer-readable storage medium, if executed by the processor, stores a computer program that causes the processor to perform the method.

[0053] Another embodiment provides a server device that provides a social media platform. The server device includes one or more processors configured to receive a selection for a base image including a human face, accept a user text prompt, and use the base image, an image mask defining areas where inpainting occurs and areas where inpainting does not occur, including at least the eye area, as input to an artificial intelligence (AI) model to generate a face decoration texture.

[0054] It will be understood that the configurations and / or approaches described herein are essentially illustrative and are subject to numerous modifications, and therefore these specific embodiments or examples should not be considered restrictively. The specific routines or methods described herein may represent one or more of any number of processing strategies. For this reason, the illustrated and / or described operations may be performed in parallel, in any other order, or omitted, in the order illustrated and / or described. Similarly, the order of the processes described above may be changed.

[0055] The subject matter of this disclosure includes novel and non-obvious combinations and subcombinations of the various processes, systems, configurations, and other features, functions, operations, and / or characteristics disclosed herein, as well as all equivalents thereof.

Claims

1. A computing system that provides a social media platform, By executing the instructions stored in the associated memory, Receive a base image containing a human face, The system receives an image mask that defines the areas where inpainting occurs and the areas where inpainting does not occur, including at least the eye area. Accepting user text prompts, In an artificial intelligence (AI) model, a face decoration texture is generated using the base image, the image mask, and the user text prompt as input. It comprises one or more processors configured to do so. Computing system.

2. The aforementioned AI model is a diffusion model. The computing system according to claim 1.

3. The aforementioned face decoration texture is a mask, and the area where the inpainting does not occur further includes the area of ​​the mouth. The computing system according to claim 1.

4. The aforementioned face decoration texture is makeup, The area where the inpainting occurs includes the area around the mouth and the area around the eyes. The region where no inpainting occurs further includes the region surrounding the mouth area. The computing system according to claim 1.

5. Receiving the base image includes receiving a selection for one of a plurality of base images. The computing system according to claim 1.

6. The one or more processors are further configured to apply the face decoration texture over human faces in a live video feed. The computing system according to claim 1.

7. The aforementioned one or more processors further operate on a pixel-by-pixel basis. Determine the hue of the skin tone of the human face in the pixel, The hue of the corresponding pixels of the face decoration texture overlaid on the human face is compared, If the difference between the hue of the face decoration texture and the hue of the skin tone is less than or equal to a threshold, the pixels of the face decoration texture are returned as they are. If the difference is greater than the threshold, the system is configured to multiply the hue of the face decoration texture by the hue of the skin tone and return the resulting value as the pixel of the face decoration texture. The computing system according to claim 6.

8. The one or more processors are further configured to present to the user of the client device a plurality of blend modes for blending the face decoration texture with the human face. The computing system according to claim 6.

9. The one or more processors are further configured to present to the user of the mobile computing device a graphical user interface (GUI) configured to not display the base image and to display the face decoration texture on top of the human face. The computing system according to claim 6.

10. The one or more processors are further configured to store the face decoration texture and make the face decoration texture available to other users of the social media platform. The computing system according to claim 1.

11. A method for social media platforms, Receiving a base image that includes a human face, The process involves receiving an image mask that defines the region where inpainting occurs and the region where inpainting does not occur, including at least the eye region. Accepting user text prompts, In an artificial intelligence (AI) model, a face decoration texture is generated using the base image, the image mask, and the user text prompt as input. A method that includes this.

12. The aforementioned AI model is a diffusion model. The method according to claim 11.

13. The aforementioned face decoration texture is a mask, and the area where the inpainting does not occur further includes the area of ​​the mouth. The method according to claim 11.

14. The aforementioned face decoration texture is makeup, The area where the inpainting occurs includes the area around the mouth and the area around the eyes. The region where no inpainting occurs further includes the region surrounding the mouth area. The method according to claim 11.

15. Receiving the base image includes receiving a selection for one of a plurality of base images. The method according to claim 11.

16. Applying the aforementioned face decoration texture to a human face in a live video feed. The method according to claim 11, further comprising:

17. In pixel units, Determining the hue of the skin tone of the human face in the pixel, Comparing the hue of the corresponding pixels of the face decoration texture overlaid on the human face, If the difference between the hue of the face decoration texture and the hue of the skin tone is less than or equal to a threshold, the pixels of the face decoration texture are returned as they are. If the difference is greater than the threshold, the hue of the face decoration texture and the hue of the skin tone are multiplied, and the resulting value is returned as the pixel of the face decoration texture. The method according to claim 16, further comprising:

18. To store the aforementioned face decoration texture and to make the aforementioned face decoration texture available to other users of the social media platform. The method according to claim 11, further comprising:

19. When executed by a processor, the processor stores a computer program that causes it to perform the method according to claim 11. A non-temporary computer-readable storage medium.

20. A server device that provides a social media platform, By executing the instructions stored in the associated memory, Receive a selection for a base image that includes a human face, Accepting user text prompts, In an artificial intelligence (AI) model, a face decoration texture is generated using the base image, an image mask defining areas where inpainting occurs and areas where inpainting does not occur (including at least the eye area), and the user text prompt as input. It comprises one or more processors configured to do so. Server device.

Images