Virtual image generation method and related equipment
By obtaining user images and 3D face models, combining stylized parameters to generate virtual images, and allowing users to adjust stylized parameters, the problem that virtual images cannot be personalized in the existing technology is solved and the user experience is improved.
Patent Information
- Application Number
- CN202310651361.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-06-02
- Publication Date
- 2024-12-03
AI Technical Summary
The prior art is difficult to generate personalized virtual images according to user preferences, and cannot adjust between "close to real faces" and "more stylized", which has poor user experience.
By obtaining the user's image and 3D face model, combining stylized parameters, generating virtual images, and previewing and adjustments in real time on the user interface, users are allowed to adjust the stylized parameters to meet personalized needs.
The ability to generate personalized virtual images based on user preferences is realized, the user experience is improved, and the styled needs of different users are met.
Smart Images

Figure CN119068157A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer technology, and in particular to a method for generating a virtual image and related equipment. Background Art
[0002] In recent years, the concept of the metaverse has become popular, and related products have appeared in the market in a spurt. "Digital people" and "virtual images" have become hot topics in this field. Among them, the virtual image generation function is introduced to the market as one of the highlights of the product. The virtual image can represent part of the user's own characteristics, and many users pursue the generation of personalized virtual images. How to generate personalized virtual images of users according to user preferences for electronic devices such as mobile phones and computers is a current and future research direction. Summary of the invention
[0003] The embodiments of the present application provide a virtual image generation method and related devices, which can generate a personalized virtual image according to user preferences and improve user experience.
[0004] In a first aspect, an embodiment of the present application provides a method for generating a virtual image, which is applied to an electronic device, and the method includes: the electronic device obtains a first image including a face; the electronic device obtains a first 3D face model; the electronic device obtains a first stylization parameter related to the first image selected by a user; the electronic device generates and displays a first virtual image based on the first image, the first 3D face model, and the first stylization parameter; the first stylization parameter indicates the proportion of influence of the first image and the first 3D face model on the first virtual image.
[0005] In the embodiment of the present application, based on the stylized parameters input by the user, the facial features of the character are fused with the stylized 3D face model to generate a virtual image, and the virtual image can be displayed on the user interface for easy browsing by the user. Since the virtual image generated is different when the user inputs different stylized parameters, the user can adjust the stylized parameters so that the generated virtual image can be adjusted between "closer to the real face" and "more stylized" to meet the preferences of different users and improve the user experience.
[0006] In some embodiments, before the electronic device obtains the first stylization parameter related to the first image selected by the user, it also includes: the electronic device generates and displays an initial virtual image based on the first image, the first 3D face model, and the default stylization parameter; the default stylization parameter indicates the proportion of influence of the first image and the first 3D face model on the initial virtual image.
[0007] In the embodiment of the present application, the developer can preset a stylization parameter (i.e., a default stylization parameter) in advance, and the electronic device can first generate an initial virtual image by fusing the facial features of the person with the stylized 3D face model based on the preset stylization parameter, and can display the initial virtual image in the user interface. The user previews the initial virtual image and then adjusts the stylization parameter until a virtual image that satisfies the user's preferences is generated, thereby improving the user experience.
[0008] In some embodiments, the electronic device generates and displays an initial virtual image based on a first image, a first 3D face model, and default stylization parameters, including: the electronic device generates a real face model based on the first image and the first 3D face model; the electronic device generates a first face model based on the real face model, the first 3D face model, and default stylization parameters; the electronic device generates an initial virtual image based on the first face model and displays a first user interface, wherein the initial virtual image is displayed in the first user interface.
[0009] In the embodiment of the present application, the real face model is actually obtained by transferring the facial features of a 2D face to the topological structure of a stylized 3D face model. The facial features of the real face model are determined by the facial features of the face in the portrait photo, and it is impossible to make the real face model contain more or less stylized features (such as more cartoon-like, more comic-like, etc.). Therefore, the real face model and the stylized 3D face model are fused based on the default stylization parameters, so that the 3D virtual image generated based on the fusion result can be adjusted between "closer to the real face" and "more stylized", meeting the preferences of different users and improving the user experience.
[0010] In some embodiments, an electronic device obtains a first image including a face, including: the electronic device displays a second user interface, the second user interface includes a first preview area and a first control, the first preview area displays an image including the face from a camera of the electronic device; the electronic device receives an operation acting on the first control, and saves the image displayed in the first preview area as the first image.
[0011] In an embodiment of the present application, a user interface may be displayed on an electronic device, and the user interface may include a display area and a first control. The display area may display character image information collected by a camera of the electronic device, and the first control may trigger the electronic device to save the character image currently collected by the camera. When the electronic device receives an operation acting on the first control, the character image currently collected by the camera may be saved as the first image to obtain the character's facial feature information, so as to facilitate the subsequent generation of a user-personalized virtual image and improve the user experience.
[0012] In some embodiments, an electronic device acquires a first image including a face, including: the electronic device displays a third user interface; the electronic device receives an operation acting on the third user interface and displays a fourth user interface, the fourth user interface displays one or more images, the one or more images include a thumbnail of the first image; the electronic device receives an operation acting on the thumbnail of the first image, and acquires the first image.
[0013] In an embodiment of the present application, a user can select an image from a gallery of an electronic device as a first image to obtain facial feature information of a person, thereby facilitating the subsequent generation of a personalized virtual image of the user and improving the user experience.
[0014] In some embodiments, before the electronic device obtains the first 3D face model, the method also includes: the electronic device displays a fifth user interface, the fifth user interface displays indication information of one or more 3D face models, the one or more 3D face models including the first 3D face model; the electronic device receives the indication information acting on the first 3D face model.
[0015] In an embodiment of the present application, multiple 3D face models can be stored in advance in the electronic device, and different 3D face models have different styles. For example, the multiple 3D face models may include a comic face model, a cartoon face model, an exaggerated face model, etc. The user can select one from the multiple 3D face models as the first 3D face model according to his or her preferences, so as to facilitate the subsequent generation of a virtual image that meets the user's preferences and enhance the user experience.
[0016] In some embodiments, the electronic device generates and displays a first virtual image based on a first image, a first 3D face model, and a first stylization parameter, including: the electronic device generates a second face model based on a real face model, the first 3D face model, and the first stylization parameter; the electronic device generates a first virtual image based on the second face model, and displays the first virtual image in a first user interface.
[0017] In an embodiment of the present application, the electronic device can, based on the user-defined degree of stylization, re-integrate the real face model with the stylized 3D face model to obtain a new virtual image, and can display the new virtual image on the electronic device, thereby achieving a real-time preview effect for the user and improving the user experience.
[0018] In some embodiments, the electronic device obtains a first stylization parameter related to a first image selected by a user, including: the electronic device receives a first operation acting on a first user interface, the first operation is used to adjust the stylization parameter; in response to the first operation, determines the first stylization parameter selected by the user.
[0019] In an embodiment of the present application, a user can freely adjust the degree of stylization (i.e., stylization parameters) on the user interface of the electronic device, and then the electronic device can, based on the user-defined degree of stylization, re-integrate the real face model with the stylized 3D face model to obtain a new virtual image, and the new virtual image can be displayed on the electronic device, thereby achieving a real-time preview effect for the user and improving the user experience.
[0020] In some embodiments, the first user interface includes a slider control, and the first operation is an operation of sliding the slider control to adjust the stylization parameter.
[0021] In an embodiment of the present application, a user can change the value of a stylized parameter by sliding a slider bar on a user interface of an electronic device, thereby obtaining a virtual image that satisfies the user's preferences, thereby improving the user experience.
[0022] In some embodiments, the method also includes: the electronic device receives an adjustment operation on the first user interface for the first virtual image, adjusts the facial features of the first virtual image, and obtains an updated first virtual image; the electronic device displays the updated first virtual image on the first user interface.
[0023] In the application embodiment, the user can also perform secondary editing on the generated virtual image to obtain a more personalized virtual image that satisfies the user's preferences, thereby improving the user experience.
[0024] In some embodiments, the electronic device generates a first virtual image based on a second face model, including: the electronic device renders the second face model to obtain the first virtual image.
[0025] In an embodiment of the present application, the generated face model can be rendered and the rendering result can be presented on the user interface of the electronic device, so that the user can browse the virtual image on the electronic device, thereby improving the user experience.
[0026] In some embodiments, the electronic device generates a first virtual image based on a second face model, including: the electronic device obtains a default 3D material, renders the second face model based on the default 3D material, and obtains the first virtual image, wherein the default 3D material includes one or more of the limb features and clothing features of the 3D character model.
[0027] In an embodiment of the present application, default 3D materials can be obtained first, such as the limb features, clothing features and other information of the 3D character model, and then the face model can be rendered based on the default 3D materials to obtain a more complete virtual image, thereby improving the user experience.
[0028] In a second aspect, an embodiment of the present application provides an electronic device, comprising a first processing unit, the first processing unit being configured to: obtain a first image comprising a face; obtain a first 3D face model; obtain a first stylization parameter associated with the first image selected by a user; generate and display a first virtual image based on the first image, the first 3D face model, and the first stylization parameter; the first stylization parameter indicating a proportion of influence of the first image and the first 3D face model on the first virtual image.
[0029] In a third aspect, an electronic device is provided, comprising: a memory and one or more processors; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to enable the electronic device to execute a method as in the first aspect or any one of the embodiments of the first aspect.
[0030] In a fourth aspect, a computer-readable storage medium is provided, comprising instructions, which, when executed on an electronic device, enable the electronic device to execute a method as in the first aspect or any one of the embodiments of the first aspect.
[0031] In a fifth aspect, a computer program product is provided. When the computer program product is run on a computer, the computer executes the method in the first aspect or any one of the embodiments of the first aspect.
[0032] In a sixth aspect, a chip system is provided, the chip system comprising at least one processor, configured to implement the method of the first aspect or any one of the embodiments of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS
[0033] Figure 1a A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.
[0034] Figure 1b A schematic diagram of the software structure of an electronic device provided in an embodiment of the present application.
[0035] Figure 1c A schematic diagram of the software structure of another electronic device provided in an embodiment of the present application.
[0036] Figure 2a-2i A set of user interfaces displayed on an electronic device provided in an embodiment of the present application.
[0037] Figure 3 A flowchart of a method for generating a virtual image provided in an embodiment of the present application.
[0038] Figure 4a-4f A schematic diagram of a set of virtual image generation method processes provided in an embodiment of the present application.
[0039] Figure 5 A flowchart of another method for generating a virtual image provided in an embodiment of the present application.
[0040] Figure 6 A schematic diagram of an application scenario of a virtual image provided in an embodiment of the present application.
[0041] Figure 7 A flowchart of another method for generating a virtual image provided in an embodiment of the present application. DETAILED DESCRIPTION
[0042] The embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.
[0043] The terms "first", "second", "third" and "fourth" etc. in the specification and claims of the present application and the drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices.
[0044] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.
[0045] The embodiment of the present application provides a method for generating a virtual image, which can be applied to electronic devices. The electronic devices may include but are not limited to smart phones, tablet personal computers (Tablet PC), handheld computers, wearable electronic devices, personal computers (PC), and desktop computers.
[0046] The virtual image generation method can be applied to games, virtual social networking, 3D model design and other scenarios.
[0047] First, several basic concepts involved in this application are introduced as follows:
[0048] Stylization: Stylization refers to caricature, cartoon or exaggeration.
[0049] Stylized base model: A 3D face model with a completely comic, cartoon or exaggerated effect, usually completed by a 3D model designer using a 3D model creation tool.
[0050] In the virtual image generation method provided in the present application, the electronic device can transfer the facial features of the face in the portrait photo to the topological structure of the stylized 3D face model to generate an initial 3D face model; then, the initial 3D face model and the stylized base model are merged to obtain the final 3D face model; finally, the final 3D face model is rendered as a 3D virtual image.
[0051] In this application, the initial 3D face model is actually obtained by migrating the facial features of a 2D face to the topological structure of a stylized 3D face model. The facial features of the initial 3D face model are determined by the facial features of the face in the portrait photo, and it is impossible to make the initial 3D face model contain more or less stylized features (such as more cartoon-like, or more comic-like, etc.). Therefore, the initial 3D face model is fused with the stylized 3D face model, so that the 3D virtual image generated based on the fusion result can be adjusted between "closer to the real face" and "more stylized", satisfying the preferences of different users and improving the user experience.
[0052] The electronic device receives a user operation to adjust the stylization degree of the 3D virtual image, and then re-merges the initial 3D face model and the stylized base model with a new stylization degree according to the user operation to obtain a new 3D face model, and then renders the new 3D face model into a new 3D virtual image.
[0053] In the present application, the user can freely adjust the degree of stylization on the user interface of the electronic device, and then the electronic device can fuse the initial 3D face model and the vertex data of the stylized base model based on the user-defined degree of stylization to obtain a new 3D virtual image, and the new 3D virtual image can be displayed on the electronic device, achieving the effect of real-time preview for the user, thereby improving the user experience.
[0054] Before introducing the method provided by the embodiment of the present application in detail, the electronic device provided by the embodiment of the present application is first introduced.
[0055] Figure 1a A schematic structural diagram of an electronic device 100 is shown.
[0056] The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a 3D camera module 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. Among them, the sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.
[0057] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.
[0058] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a central processing unit (CPU), a graphics processing unit (GPU), a neural-network processing unit (NPU), a modem processor, an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, etc. Among them, different processing units may be independent devices or integrated in one or more processors. In some embodiments, the electronic device 100 may also include one or more processors 110.
[0059] The controller may be the nerve center and command center of the electronic device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.
[0060] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or circulated. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the electronic device 100.
[0061] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.
[0062] Among them, the MIPI interface can be used to connect the processor 110 with peripheral devices such as the display screen 194 and the 3D camera module 193. The MIPI interface includes a camera serial interface (CSI), a display serial interface (DSI), etc. In some embodiments, the processor 110 and the 3D camera module 193 communicate through the CSI interface to realize the camera function (also known as the photo function) of the electronic device 100. The processor 110 and the display screen 194 communicate through the DSI interface to realize the display function of the electronic device 100.
[0063] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present application is only a schematic illustration and does not constitute a structural limitation on the electronic device 100. In other embodiments, the electronic device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.
[0064] The electronic device 100 can realize the display function through the GPU, the display screen 194, and the application processor. The GPU is a microprocessor for image processing, which is connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and is used for graphics rendering, such as rendering a virtual image displayed on the display screen 194. The processor 110 may include one or more GPUs, which execute instructions to generate or change display information.
[0065] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.
[0066] The electronic device 100 can realize the camera function through a 3D camera module 193, an ISP, a video codec, a GPU, a display screen 194, an application processor AP, a neural network processor NPU, etc.
[0067] The 3D camera module 193 can be used to collect color image data of the photographed object. The ISP can be used to process the color image data collected by the 3D camera module 193. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing and converts it into an image visible to the naked eye. The ISP can also perform algorithm optimization on the noise and brightness of the image. The ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, the ISP can be set in the 3D camera module 193.
[0068] In some embodiments, the 3D camera module 193 may be composed of a color camera module.
[0069] In some embodiments, the photosensitive element of the camera of the color camera module can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, and then transmits the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format.
[0070] In some embodiments, the electronic device 100 may include one or N 3D camera modules 193, where N is a positive integer greater than 1. Specifically, the electronic device 100 may include one front 3D camera module 193 and one rear 3D camera module 193. The front 3D camera module 193 may generally be used to collect color image data of the photographer facing the display screen 194, and the rear 3D camera module 193 may be used to collect color image data of the subject (such as a person, scenery, etc.) facing the photographer.
[0071] In some embodiments, the CPU or GPU or NPU in the processor 110 can process the color image data collected by the 3D camera module 193. In some embodiments, the NPU can identify the color image data collected by the 3D camera module 193 (specifically the color camera module) through a neural network algorithm based on the skeleton point recognition technology, such as a convolutional neural network algorithm (CNN), to determine the skeleton points of the photographed person. The CPU or GPU can also run a neural network algorithm to determine the skeleton points of the photographed person based on the color image data. In some embodiments, the CPU or GPU or NPU can also be used to migrate the facial features of the face in the portrait photo to the topological structure of the stylized 3D face model to generate an initial 3D face model; then, the initial 3D face model and the stylized base model are fused to obtain the final 3D face model; finally, the final 3D face model is rendered as a 3D virtual image. This will be described in detail later, so I will not go into details here.
[0072] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, applications such as intelligent cognition of electronic device 100 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.
[0073] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function. For example, data such as music, photos, and videos can be stored in the external memory card.
[0074] The internal memory 121 can be used to store one or more computer programs, which include instructions. The processor 110 can run the above instructions stored in the internal memory 121, so that the electronic device 100 executes the virtual image generation method provided in some embodiments of the present application, as well as various functional applications and data processing. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area can store an operating system; the program storage area can also store one or more applications (such as a gallery, contacts, etc.). The data storage area can store data (such as photos, etc.) created during the use of the electronic device 100. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.
[0075] Figure 1a The electronic device 100 shown in the example can display the various user interfaces described in the following embodiments through the display screen 194. The electronic device 100 can detect touch operations in various user interfaces through the touch sensor 180K, such as click operations in various user interfaces (such as touch operations on icons, double-click operations), and for example, upward or downward sliding operations in various user interfaces, or operations of performing circle gestures, etc. In some embodiments, the electronic device 100 can detect motion gestures performed by the user holding the electronic device 100, such as shaking the electronic device, through a gyroscope sensor 180B, an acceleration sensor 180E, etc. In some embodiments, the electronic device 100 can detect non-touch gesture operations through a 3D camera module 193 (such as a 3D camera).
[0076] The software system of the electronic device 100 may adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present application takes the Android system of the layered architecture as an example to exemplify the software structure of the electronic device 100.
[0077] Figure 1b It is a software structure block diagram of the electronic device 100 according to an embodiment of the present application.
[0078] The layered architecture divides the software into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom, namely, the application layer, the application framework layer, the Android runtime (Android runtime) and the system library, and the kernel layer.
[0079] The application layer can include a series of application packages.
[0080] like Figure 1b As shown, the application package may include smart photo taking application, camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message and other applications.
[0081] Among them, the smart camera application can be used to generate a user's virtual image. The smart camera application can receive user interactive operations on the graphical interface, such as the user uploading a portrait photo, the user adjusting the stylized parameters, the user manually pinching the face, etc. The smart camera application can be a system application or a third-party application. The functions of the smart camera application have been clearly introduced here and in the subsequent method embodiments, and its name does not constitute a limitation on the present application. For example, the smart camera application can also be called "3D pinching face", "3D character construction", etc., which are not limited here.
[0082] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some predefined functions.
[0083] like Figure 1b As shown, the application framework layer may include a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, and the like.
[0084] The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc.
[0085] Content providers are used to store and retrieve data and make it accessible to applications. The data may include images, videos, audio, calls made and received, browsing history and bookmarks, phone books, etc.
[0086] The view system includes visual controls, such as controls for displaying text, controls for displaying images, etc. The view system can be used to build applications. A display interface can be composed of one or more views. For example, a display interface including a text notification icon can include a view for displaying text and a view for displaying images.
[0087] The phone manager is used to provide communication functions of the electronic device 100, such as management of call status (including connecting, hanging up, etc.).
[0088] The resource manager provides various resources for the application, such as localized images, strings, icons, layout files, video files, and so on.
[0089] The notification manager enables applications to display notification information in the status bar. It can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction. For example, the notification manager is used to notify download completion, message reminders, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or scroll bar text, such as notifications of applications running in the background, or a notification that appears on the screen in the form of a dialog window. For example, a text message is displayed in the status bar, a prompt sound is emitted, an electronic device vibrates, an indicator light flashes, etc.
[0090] Android Runtime includes core libraries and virtual machines. Android runtime is responsible for scheduling and management of the Android system.
[0091] The core library consists of two parts: one part is the function that needs to be called by the Java language, and the other part is the Android core library.
[0092] The application layer and the application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.
[0093] The system library may include multiple functional modules, such as surface manager, media library, 3D engine module (such as OpenGL ES), 2D graphics engine (such as SGL), model generation module, etc.
[0094] The surface manager is used to manage the display subsystem and provide the fusion of 2D and 3D layers for multiple applications.
[0095] The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, G.264, MP3, AAC, AMR, JPG, PNG, etc.
[0096] 3D engine module, used to implement three-dimensional graphics drawing, image rendering, synthesis, and layer processing, etc.
[0097] A 2D graphics engine is a drawing engine for 2D drawings.
[0098] The model generation module is used to receive the portrait photo and stylization parameters input by the user, and output the stylized 3D face pinching result.
[0099] The kernel layer is the layer between hardware and software. The kernel layer contains at least display driver, camera driver, audio driver, and sensor driver.
[0100] In some embodiments, the combination Figure 1c , detailing the model generation module and 3D engine module in this application.
[0101] The model generation module (Avatar SDK) can be used to receive the portrait image and stylization parameters input by the user, and output the stylized face model. The Avatar SDK may include three functional units, namely the real 3D face reconstruction module, the vertex data weighted fusion module, and the stylized base model topology migration module. Among them, the real 3D face reconstruction module can be input as a portrait image, and output a real face model through a 3D face reconstruction algorithm. Based on the real face model, the vertex data of the real face model can be obtained, and the vertex data of the real face model can be cached, so that the user can achieve the purpose of real-time adjustment and preview when adjusting the stylization parameters. The vertex data weighted fusion module can be input as the vertex data of the real face model, the vertex data of the target stylized base model, and the stylized parameters set by the user / developer. The stylized 3D face model is output by weighted fusion of vertex data. The stylized base model topology migration module relies on the topological structure of the target stylized base model. Through the deformation algorithm, the topological structure of the real face model generated by the real 3D face reconstruction module is made consistent with the topological structure of the target stylized base model, that is, the vertex data of the real face model corresponds one-to-one to the vertex data of the target stylized base model.
[0102] The 3D engine module is used to parse the results output by the Avatar SDK, and in response to the update of the manual adjustment results by the user, perform real-time rendering and frame output, and feed back the rendering results (i.e., the virtual image) to the smart camera application to allow the user to preview the virtual image effect. In some embodiments, the 3D engine module can perform real-time rendering and frame output of the vertex data of the stylized 3D face model output by the Avatar SDK, thereby enabling a virtual image preview in the smart camera application interface. In some embodiments, the 3D engine module can obtain manual adjustment information input by the user from the smart camera application, which may include but is not limited to bone coefficients, material index values, etc., and perform real-time rendering and frame output based on the vertex data and manual adjustment information of the stylized 3D face model output by the Avatar SDK, thereby enabling a virtual image preview in the smart camera application interface.
[0103] The following introduces the user interface provided by the embodiments of the present application.
[0104] Figure 2a-2i A set of user interfaces displayed on the electronic device 100 provided in an embodiment of the present application.
[0105] Figure 2a An exemplary user interface 40 for displaying installed applications on the electronic device 100 is exemplarily shown.
[0106] like Figure 2a As shown, the user interface 40 displays one or more installed application icons, such as the user interface 40 displays smart photography 301, gallery 302, clock, calendar, memo, file management, email, music, calculator, Huawei video, sports health, weather, browser, smart life, settings, and recorder.
[0107] like Figure 2a As shown, the electronic device 100 can receive and respond to the user's touch operation on the smart camera 301 icon of the user interface 40 (i.e., the operation of opening the smart camera application), start the smart camera application, and display the following information on the display screen 194 of the electronic device 100: Figure 2b The user interface 41 shown may include a create character control 303 .
[0108] like Figure 2b As shown, the electronic device 100 can receive and respond to the user's touch operation on the create character control 303 of the user interface 41, and then display the following on the display screen 194: Figure 2cThe user interface 42 shown may include a target display area 304, a photo control 305, and a template display area 306. The target display area 304 may be used to display a stylized template selected by a user; and the template display area 306 may be used to display indication information of multiple selectable stylized templates.
[0109] like Figure 2c As shown, the electronic device 100 can receive and respond to the user's operation of selecting a target stylized template from multiple stylized templates. For example, if the user touches the indication information of the first stylized template in the template display area 306, the electronic device 100 can receive and respond to the user's touch operation on the indication information of the first stylized template, and the electronic device 100 can determine the first stylized template as the target stylized template.
[0110] In some embodiments, user interface 42 may be a fifth user interface.
[0111] like Figure 2d As shown, the electronic device 100 may display a target stylized template in the target display area 304 of the user interface 42 .
[0112] like Figure 2d As shown, the user can touch the photo control 305 on the user interface 42, and the electronic device 100 can receive and respond to the user's touch operation on the photo control 305, start the camera of the electronic device 100, and display the following on the display screen 194: Figure 2e The user interface 43 shown may include a picture upload control 307, a photo taking control 308 and a preset display area 309. The preset display area 309 may be used to display the person image captured by the camera in real time.
[0113] In some embodiments, the user interface 43 may be a second user interface, or the user interface 43 may be a third user interface.
[0114] like Figure 2e As shown, the electronic device 100 receives and responds to the user's touch operation on the photo control 308 of the user interface 43, saves the character image captured by the current camera, and determines the character image as the first image. The first image includes real human face features, and the real human face features may include the face shape features, facial features, etc. of the character.
[0115] like Figure 2e As shown, the electronic device 100 receives and responds to the user's touch operation on the upload picture control 307 of the user interface 43, the electronic device 100 starts the gallery 302 application, and displays on the display screen 194 as shown in FIG. Figure 2fThe user interface 44 shown can be used to display thumbnails of multiple pictures stored in the electronic device 100. Further, the electronic device 100 can receive and respond to the user's operation of selecting a target picture from multiple pictures, such as the user's operation of selecting the thumbnail of the first picture in the gallery, and then the electronic device 100 can use the first picture as the first image, and the first image includes real facial features, and the real facial features can include the face shape features, facial features, etc. of the person.
[0116] In some embodiments, user interface 44 may be a fourth user interface.
[0117] In some embodiments, after the electronic device 100 obtains the first image, the default stylization parameters and the target stylization template, it can generate an initial virtual image based on the first image, the default stylization parameters and the target stylization template. How to generate the initial virtual image will be described in detail later, and will not be repeated here. After the electronic device 100 generates the initial virtual image, it can display the image on the display screen 194 as shown in FIG. Figure 2g The user interface 45 shown in the figure may include an initial virtual image and a preset control 310 for adjusting the stylized parameters. The preset control for adjusting the stylized parameters may include but is not limited to a progress bar, an input box, a disc-type sliding button, Figure 2g Take the progress bar as an example.
[0118] In some embodiments, Figure 2g It can be the first user interface.
[0119] It should be noted that the stylization parameter can be understood as the ratio of the target stylized master model features to the virtual image features, and the stylization parameter can be continuously adjusted in the range of [0, 100%]. The default stylization parameter can be a value preset by the developer in advance. The virtual image can be understood as a 3D face model obtained by fusing the real face features with the target stylized master model features according to the stylization parameters. For example, the face shape features of the virtual image are obtained by fusing the face shape features of the character with the face shape features of the target stylized master model according to the stylization parameters. The initial virtual image can be understood as a 3D face model obtained by fusing the real face features with the target stylized master model features according to the default stylization parameters.
[0120] like Figure 2gAs shown, the electronic device 100 can receive and respond to the user's operation on the preset control 310 of the user interface 45 to determine the user-defined stylization parameters. If the preset control 310 is a progress bar, the user operation can be a sliding progress bar operation; if the preset control 310 is an input box, the user operation can be a user inputting a stylization parameter value; if the preset control 310 is a disc-type sliding button, the user operation can be sliding the disc-type sliding button. Further, the electronic device 100 generates an updated virtual image based on the user-defined stylization parameters, the first image and the target stylization template, and displays the updated virtual image on the user interface 45. How to regenerate the virtual image will be described in detail later, so it will not be repeated here.
[0121] In some embodiments, the user's operation on the preset control 310 of the user interface 45 may be the first operation.
[0122] like Figure 2h As shown, the user can slide on the preset control 310 (taking the progress bar as an example) of the user interface 45. Different positions on the progress bar correspond to different stylized parameters. When the user slides the progress bar to the left, the stylized parameter decreases. When the user slides the progress bar to the right, the stylized parameter increases. Figure 2h (a) The user slides to 10%, indicating that the stylization parameter is 10%; Figure 2h (b) The user slides to 50%, indicating that the stylization parameter is 50%; Figure 2h In (c), the user slides to 100%, indicating that the stylization parameter is 100%.
[0123] In this application, if Figure 2h As shown, the user can arbitrarily adjust the stylization parameters on the user interface 45, and the virtual image based on different stylization parameters can be displayed in real time on the user interface 45, so that the user can freely adjust between the real face proportions and the stylized face proportions, thereby improving the user experience.
[0124] like Figure 2i As shown, the user interface 45 also includes a display area 311 and a display area 312, and the display area 311 includes multiple local feature controls, such as eyebrow controls, nose controls, clothing controls, headwear controls, etc. The user can view the local feature materials by clicking on the local feature controls. For example, if the user clicks on the eyebrow control, the electronic device can display multiple materials about local features (such as materials of multiple eyebrows) on the display area 312, and then the user can select one from the multiple materials, such as selecting the first eyebrow, and then the electronic device can adjust the eyebrow shape of the virtual image according to the shape of the first eyebrow, and generate an updated virtual image again.
[0125] Next, we will combine the attached Figure 3 and Figure 4a-4f The virtual image generation method provided by the embodiment of the present application is described in detail as follows:
[0126] S101, the electronic device starts a first application.
[0127] Specifically, the first application can be used to generate a virtual image for the user. The first application can be the above Figure 1b Smart camera applications in . Smart camera applications can be system applications or third-party applications.
[0128] Exemplarily, the electronic device 100 may receive and respond to a user's touch operation on the smart camera 301 icon of the user interface 40 (ie, an operation to open the smart camera application), and the electronic device 100 launches the first application.
[0129] S102: Acquire a first image including a person through a first application.
[0130] Specifically, the first image includes real facial features, which may include two-dimensional facial features, two-dimensional facial features, etc. The first image may be one or more pictures, one or more animated pictures, or one or more videos. The first image may be obtained by real-time shooting of an electronic device, or may be obtained from a gallery stored in the electronic device.
[0131] In some embodiments, Figure 2e As shown, the electronic device 100 can receive and respond to the user's touch operation on the photo control 308 of the user interface 43, save the character image captured by the current camera, and determine the character image as the first image.
[0132] In other embodiments, Figure 2e As shown, the electronic device 100 receives and responds to the user's touch operation on the upload picture control 307 of the user interface 43, the electronic device 100 starts the gallery 302 application, and displays on the display screen 194 as shown in FIG. Figure 2f The user interface 44 shown may be used to display thumbnails of multiple pictures stored in the electronic device 100. Further, the electronic device 100 may receive and respond to an operation of a user selecting a target picture from multiple pictures, such as an operation of a user selecting a thumbnail of the first picture in the gallery, and the electronic device 100 may use the first picture as the first image.
[0133] S103: Determine the first 3D base model selected by the user.
[0134] Specifically, the first 3D base model (also referred to as the first 3D face model) can be the target stylized base model mentioned above, that is, a stylized 3D character model designed by a modeler, such as a standard face shape / facial features that is cartoonized, comic-like, or exaggerated in design and production by a modeler.
[0135] In some embodiments, the electronic device may display a user interface provided by the first application and displaying a plurality of primitives, detect a user operation of selecting a primitive on the user interface, and determine the first 3D primitive selected by the user according to the user operation. Figure 2c As shown, the electronic device 100 can receive and respond to the user's operation of selecting a target stylized template from multiple stylized templates. For example, if the user touches the indication information of the first stylized template in the template display area 306, the electronic device 100 can receive and respond to the user's touch operation on the indication information of the first stylized template, and the electronic device 100 can determine the first stylized template as the target stylized template (i.e., the first 3D template).
[0136] In some embodiments, S103 may be performed first and then S102.
[0137] Optionally, S104, the electronic device may display a user interface for displaying 3D materials provided by the first application, detect a user operation of selecting a material on the user interface, and determine the 3D material selected by the user according to the user operation. The 3D material includes but is not limited to hairstyles, clothing, jewelry, etc.
[0138] S105, generating a real face model according to the first image and the topological structure of the first 3D base model.
[0139] Specifically, the real face model is actually obtained by transferring the facial features of the face in the first image to the topological structure of the first 3D base model. The facial features of the face in the real face model are determined by the facial features of the face in the first image.
[0140] First, the first image is preprocessed to obtain a face area image.
[0141] Specifically, Figure 4a As shown, the electronic device can pre-process the first image (i.e., the portrait image) input by the user, such as detecting the face area in the photo through a face detection algorithm. Then, the face key points of the area are extracted, and the deflection angle of the face is normalized and corrected through a face alignment algorithm, so that the aligned face area image can be output. Among them, the face area image includes face shape features, facial features, etc. that are closer to the real facial features of the person.
[0142] Then, the face area image is processed based on the first 3D base model and a preset algorithm to obtain a real face model.
[0143] Specifically, Figure 4b As shown, the electronic device can process the face area image based on the classic 3D deformable face model (3DMorphable Models, 3DMM) technology principle, that is, reconstruct the real 3D face, obtain the real face model, and cache the data. In order to make the generated real face model consistent with the topological structure of the first 3D base model, a non-rigid deformation algorithm can be used to perform topological migration processing on the 3DMM data.
[0144] It should be noted that the topological structure of the 3D model (such as the first 3D base model, or the real face model) represents the layout, structure, and connection of points, lines, and surfaces in the 3D model. Since the real face model and the target stylized base model have the same topological structure, the facial semantic information expressed by the points, lines, and surfaces in the real face model (for example, the point, line, and surface index values corresponding to the nose area) corresponds one-to-one to the target stylized base model, such as the vertex data of the real face model corresponds one-to-one to the vertex data of the target stylized base model.
[0145] It should also be noted that the vertex data of the target stylized base model includes the face shape features, facial features (such as forehead, eyebrows, eyes, nose, lips, etc.) of the target stylized base model. The vertex data of the real face model includes the real facial features of the person, such as face shape features, facial features, etc.
[0146] S106: Generate a first face model according to the real face model and the first 3D base model based on default stylization parameters.
[0147] Specifically, the stylization parameter is the ratio of the first 3D base model feature to the virtual image feature, and the stylization parameter can be continuously adjusted within the range of [0, 100%]. The default stylization parameter can be a value preset in advance by the developer. The first face model can indicate the vertex data of the initial virtual image. The facial features of the face of the first face model are between the facial features of the face of the real face model and the facial features of the face of the first 3D base model. The initial virtual image can be the result presented after the electronic device renders based on the real face model and the first 3D base model, and is used to display the eyes, nose, mouth, skin and other features of the stylized character on the user interface.
[0148] In some embodiments, the first 3D face model indicates a target stylized base model, and the first face model indicates an initial 3D face model generated based on the first image and the target stylized base model.
[0149] For example, Figure 4cAs shown, based on the vertex data of the real face model and the vertex data of the target stylized base model (i.e., the first 3D base model), the vertex data is weightedly fused according to the default stylization parameters, and the 3D face structure (i.e., the first face model) of the default stylization parameters is output. The stylization parameters are used as weight values to balance the proportion of real face and stylized face features. The vertex data weighted fusion formula is shown in Formula 1.
[0150] Mesh stylized =w*Mesh target +(1-w)*Mesh real (Formula 1)
[0151] Mesh target Represents the vertex data of the target stylized primitive, Mesh real Represents the vertex data of the real face model, Mesh stylized represents vertex data of a stylized 3D face model (such as the first face model), and w represents a stylization parameter.
[0152] S107, generating an initial virtual image according to the first face model.
[0153] Specifically, vertex data is obtained according to the first face model, parsed and loaded into the 3D scene, and the vertex data is used to replace the facial vertex data of a default virtual image in the 3D scene (such as the virtual image corresponding to the first 3D model) to obtain an initial virtual image. The initial virtual image can use the bone parameters, expression base parameters, and texture material properties of the default virtual image.
[0154] In some embodiments, the vertex data of the first face model is parsed and loaded into the 3D scene, replacing the face vertex data of the default virtual image (such as the virtual image corresponding to the first 3D model) in the 3D scene. At the same time, based on the 3D material input by the user, the hairstyle, clothing, jewelry, etc. of the virtual image are determined from the 3D material library to generate an initial virtual image. The 3D material library stores multiple hairstyle models, clothing models, jewelry models, etc. of the virtual image.
[0155] For example, Figure 2g and Figure 4d As shown, the electronic device can render frames in real time and display the initial virtual image on the display screen 194 for the user to preview, so that the user can view the generated personalized virtual image in real time, thereby improving the user experience.
[0156] It should be noted that, the higher the default stylization parameter setting is, the higher the stylization degree of the initial virtual image is (such as the higher the cartoon degree is), that is, the initial virtual image is more similar to the target stylized base model; the lower the default stylization parameter setting is, the lower the stylization degree of the initial virtual image is (such as the lower the cartoon degree is), that is, the initial virtual image is more similar to the real face model.
[0157] In this application, the real face model is actually obtained by migrating the facial features of a 2D human face to the topological structure of the first 3D base model. The facial features of the real face model are determined by the facial features of the human face in the portrait photo, and it is impossible to make the real face model contain more or less stylized features (such as more cartoon-like, more comic-like, etc.). Therefore, the real face model is fused with the first 3D base model, so that the virtual image generated based on the fusion result can be adjusted between "closer to the real face" and "more stylized", meeting the preferences of different users and improving the user experience.
[0158] S108: receiving a user operation for adjusting a stylization parameter, and determining a new stylization parameter according to the user operation.
[0159] Specifically, the user operations used to adjust the stylized parameters can be of various types, such as operations acting on the user interface (touch operations, drag operations), voice commands, air operation commands, etc., which are not specifically limited in this application.
[0160] For example, Figure 2g As shown, the electronic device 100 can receive and respond to the user's sliding operation on the progress bar (i.e., the preset control 310) of the user interface 45 to determine the user-defined stylization parameters, i.e., the new stylization parameters. Figure 2h As shown, the user can perform dynamic operations on the progress bar of the user interface 45. Different positions on the progress bar correspond to different stylized parameters. When the user slides the progress bar to the left, the stylized parameter decreases. When the user slides the progress bar to the right, the stylized parameter increases. For details, please refer to the above Figure 2h The relevant description will not be repeated here.
[0161] S109, generating a second face model based on the real face model and the first 3D base model and the new stylized parameters.
[0162] Specifically, the vertex data of the real face model and the vertex data of the first 3D base model can be obtained, and the new stylization parameters are used as weight values. The vertex data of the real face model and the vertex data of the first 3D base model are re-weighted and fused through the above formula to obtain a second face model, and the second face model can indicate the vertex data of the updated virtual image. The facial features of the face of the second face model are between the facial features of the face of the real face model and the facial features of the face of the first 3D base model. The difference between the second face model and the first face model is that the proportion of the first 3D base model features is different, so the corresponding virtual image is also different.
[0163] For example, Figure 4e As shown, the user can observe the preview effect of the initial virtual image and further select the virtual image in the user interface (such as Figure 2g The sliding bar on the user interface 45 shown in the figure modifies the stylization parameters to balance the stylization ratio of the virtual image. After receiving the new stylization parameters, the electronic device re-merges the real face model with the target style base model and refreshes and outputs a new face model.
[0164] It should be noted that when the slider is dragged to the right, the stylization parameter increases, and the stylization degree of the updated virtual image is higher (such as the higher the cartoon degree), that is, the updated virtual image is more similar to the target stylized base model; when the slider is dragged to the left, the stylization parameter decreases, and the stylization degree of the updated virtual image is lower (such as the lower the cartoon degree), that is, the updated virtual image is more similar to the real face model.
[0165] S110, generating an updated virtual image according to the second face model.
[0166] Specifically, the vertex data of the second face model is parsed and loaded into the 3D scene, and the facial vertex data of the default virtual image in the 3D scene (such as the virtual image corresponding to the first 3D model) is replaced to obtain an updated virtual image. The updated virtual image can use the bone parameters, expression base parameters, and texture material properties of the default virtual image.
[0167] In some embodiments, the updated virtual image may be the first virtual image.
[0168] In some embodiments, the vertex data of the second face model is parsed and loaded into the 3D scene, replacing the face vertex data of the default virtual image (such as the virtual image corresponding to the first 3D model) in the 3D scene. At the same time, based on the 3D material input by the user, the hairstyle, clothing, jewelry, etc. of the virtual image are determined from the 3D material library, thereby generating an updated virtual image.
[0169] For example, Figure 4eAs shown, the electronic device can display the updated virtual image in real time in the user interface. The user can input a single face photo and stylization parameters, the system calls the algorithm to generate a 3D face model, and then fuses it with the target stylization base model according to the stylization parameters to obtain a stylized 3D face model, and then parses and loads the result, and renders the frame to the user interface for preview. Furthermore, the user can adjust and modify the stylization parameters according to the preview results. After receiving the new stylization parameters, the system will fuse the 3D face model with the target style base model again, refresh and output a new stylized 3D face model, and feed it back to the user interface in real time, thereby generating a personalized virtual image that meets the user's preferences and improving the user experience.
[0170] In this application, if Figure 4f As shown, the user can adjust the stylization parameters by sliding, and can preview the virtual image with different stylization deformation degrees in real time. Specifically, the user can freely adjust the stylization parameters on the user interface of the electronic device, and then the electronic device can fuse the real face model with the first 3D base model based on the user-defined stylization parameters to obtain a new 3D virtual image, and the new 3D virtual image can be displayed on the electronic device, achieving the effect of real-time preview for the user, thereby improving the user experience.
[0171] Optional S111, receiving a user operation for adjusting facial features, and adjusting the facial features of the updated virtual image.
[0172] Specifically, the electronic device may receive and respond to the user's manual adjustment operation (i.e., the user's operation of adjusting facial features), and may obtain manual adjustment information, which includes but is not limited to updated bone coefficients, updated expression base coefficients, updated dress and accessories models, updated texture material properties, etc. Then, the electronic device may generate an updated virtual image again based on the second face model and the manual adjustment information.
[0173] For example, Figure 2i As shown, the user interface 45 also includes a display area 311 and a display area 312, and the display area 311 includes multiple local feature controls, such as eyebrow controls, nose controls, clothing controls, headwear controls, etc. The user can view the local feature materials by clicking on the local feature controls. For example, if the user clicks on the eyebrow control, the electronic device can display multiple materials about local features (such as materials of multiple eyebrows) on the display area 312, and then the user can select one from the multiple materials, such as selecting the first eyebrow, and then the electronic device can adjust the eyebrow shape of the virtual image according to the shape of the first eyebrow, and generate an updated virtual image again.
[0174] The electronic device can display a manual adjustment and status interface, which can be used to manually adjust the local position of the virtual image's face through the bones and expression bases bound in advance by the modeler. The modeler can also customize the appearance (including hairstyle, hair color, glasses, clothes, etc.) from the material library prepared in advance.
[0175] In this application, users can also perform secondary editing on the generated 3D virtual image on the user interface, such as manually adjusting the eyebrows, nose, mouth, face shape, etc. of the 3D virtual image, to generate a more personalized virtual image that meets the user's preferences.
[0176] Reference Figure 5 , Figure 5 The process of how various modules inside the electronic device cooperate to realize the virtual image method is introduced.
[0177] S201, start a smart camera application, the smart camera application receives a first image uploaded or photographed by a user, and receives a first 3D base model selected by the user.
[0178] After the electronic device receives the user's operation to open the smart camera application, the smart camera application is started. For the specific implementation of S201 and S202, please refer to the previous article. Figure 3 A detailed introduction to S101.
[0179] S202, the smart camera application receives and responds to a first request initiated by a user, and sends a first image to a model generation module.
[0180] In some embodiments, the first request includes a first 3D base model.
[0181] S203: The model generation module preprocesses the first image to obtain a face area image.
[0182] Among them, the specific implementation of S203 can refer to the previous article Figure 3 Detailed introduction of S105.
[0183] S204, the model generation module generates a real face model according to the face area image and the topological structure of the first 3D base model.
[0184] Among them, the specific implementation of S203 can refer to the previous article Figure 3 Detailed introduction of S105.
[0185] S205: The model generation module determines and saves vertex data of the real face model.
[0186] S206, the model generation module generates a first face model according to the vertex data of the real face model and the vertex data of the first 3D base model based on default stylization parameters.
[0187] The specific implementation of S206 can be found in the previous article. Figure 3 Detailed introduction of S106.
[0188] S207: The model generation module sends the first face model to the 3D engine module.
[0189] S208, the 3D engine module renders a frame for the first face model to obtain an initial virtual image.
[0190] S209, the 3D engine module sends an initial virtual image to the smart camera application.
[0191] S210, the smart camera application displays an initial virtual image through a display screen of the electronic device.
[0192] For the specific implementation of S207-S210, please refer to the previous article. Figure 3 Detailed introduction of S107.
[0193] S211: The smart camera application receives a user operation for adjusting a stylization parameter and determines the stylization parameter.
[0194] Among them, the user can in the user interface (such as Figure 2g The stylized parameters are adjusted on the user interface shown in FIG. 1 , and then the smart photo taking application executes S211.
[0195] S212, the smart camera application sends stylized parameters to the model generation module.
[0196] S213, the model generation module determines new stylization parameters; generates a second face model based on the new stylization parameters according to the vertex data of the real face model and the vertex data of the first 3D base model.
[0197] Among them, the specific implementation of S211-S213 can refer to the previous article Figure 3 Detailed introduction of S108 and S109.
[0198] S214: The model generation module sends the second face model to the 3D engine module.
[0199] S215, the 3D engine module renders a frame of the second face model to obtain an updated virtual image.
[0200] S216, the 3D engine module sends the updated virtual image to the smart camera application.
[0201] S217, the smart camera application displays the updated virtual image through the display screen of the electronic device.
[0202] Among them, the specific implementation of S214-S217 can refer to the previous article Figure 3 A detailed introduction to S110.
[0203] S218, the smart camera application sends face adjustment information to the 3D engine module.
[0204] Among them, the user can initiate a user operation of adjusting facial features on the user interface, and then the smart camera application can execute S218.
[0205] S219, the 3D engine module adjusts the facial features of the updated virtual image based on the facial adjustment information.
[0206] S220: The 3D engine module sends the adjusted virtual image to the smart camera application.
[0207] S221, the smart camera application displays the adjusted virtual image through a display screen of the electronic device.
[0208] Among them, the specific implementation of S218-S221 can refer to the previous article Figure 3 Detailed introduction of S111.
[0209] In some embodiments, the virtual image obtained by the virtual image generation method provided by the present application can be applied to Figure 6 In the games, virtual social networking, 3D model design and other scenes shown. For example, the characters in the game can be replaced by the virtual images obtained by this application, and the characters in the virtual social networking can be replaced by the virtual images obtained by this application.
[0210] Please see attached Figure 7 Next, we will combine the attached Figure 7 The process of another virtual image generation method provided by the embodiment of the present application is introduced, and is described in detail as follows:
[0211] S301: The electronic device obtains a first image including a face.
[0212] For the specific implementation of S301, please refer to the previous article. Figure 3 Detailed introduction of S102.
[0213] S302: The electronic device obtains a first 3D face model.
[0214] Among them, the specific implementation of S302 can refer to the previous article Figure 3 Detailed introduction of S103.
[0215] S303: The electronic device obtains a first stylization parameter related to the first image selected by the user.
[0216] The specific implementation of S303 can be found in the previous article. Figure 3 Detailed introduction of S108.
[0217] S304: The electronic device generates and displays a first virtual image according to the first image, the first 3D face model, and the first stylization parameter.
[0218] Specifically, the first stylization parameter indicates the influence ratio of the first image and the first 3D face model on the first virtual image. Figure 3 Detailed introduction of S105, S109 and S110.
[0219] In the embodiment of the present application, based on the stylized parameters input by the user, the facial features of the character are fused with the stylized 3D face model to generate a virtual image, and the virtual image can be displayed on the user interface for easy browsing by the user. Since the virtual image generated is different when the user inputs different stylized parameters, the user can adjust the stylized parameters so that the generated virtual image can be adjusted between "closer to the real face" and "more stylized" to meet the preferences of different users and improve the user experience.
[0220] In some embodiments, the electronic device obtains default stylization parameters, and generates N initial virtual images based on the first image, the default stylization parameters and N 3D primitives (N is an integer greater than or equal to 1), and displays the N initial virtual images on the display screen 194 of the electronic device. Further, the electronic device receives and responds to the user's operation of selecting a target initial virtual image from the N initial virtual images, and displays the target initial virtual image and a progress bar for adjusting the stylization parameters on the display screen 194. Then, the electronic device can receive and respond to the user's sliding operation on the progress bar, determine new stylization parameters, and regenerate the virtual image based on the new stylization parameters, the first image and the target stylization primitive (i.e., the 3D primitive corresponding to the target initial virtual image), and display the regenerated virtual image on the display screen 194.
[0221] The present application provides an electronic device, which includes a first processing unit, and the first processing unit is used to: obtain a first image including a face; obtain a first 3D face model; obtain a first stylization parameter related to the first image selected by a user; generate and display a first virtual image according to the first image, the first 3D face model, and the first stylization parameter; the first stylization parameter indicates the influence ratio of the first image and the first 3D face model on the first virtual image.
[0222] The present application provides a computer storage medium, characterized in that the computer storage medium stores a computer program, and when the computer program is executed by a processor, any one of the above-mentioned virtual image generation methods is implemented.
[0223] An embodiment of the present application provides an electronic device, the electronic device includes a processor, and the processor is configured to support the electronic device to implement the corresponding functions in any of the above-mentioned virtual image generation methods. The electronic device may also include a memory, which is coupled to the processor and stores the necessary program instructions and data of the electronic device. The electronic device may also include a communication interface for the electronic device to communicate with other devices or a communication network.
[0224] The present application provides a chip system, which includes a processor for supporting an electronic device to implement the functions involved above, for example, generating or processing information involved in the above-mentioned virtual image generation method. In a possible design, the chip system also includes a memory, which is used to store program instructions and data necessary for the electronic device. The chip system can be composed of a chip, or it can include a chip and other discrete devices.
[0225] The present application provides a computer program, characterized in that the computer program includes instructions, and when the computer program is executed by a computer, the computer executes the above-mentioned virtual image generation method.
[0226] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0227] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the order of the actions described, because according to the present application, some steps may be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.
[0228] In the several embodiments provided in the present application, it should be understood that the disclosed device can be implemented in other ways. For example, the device embodiments described above are only schematic, such as the division of the above-mentioned units, which is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical or other forms.
[0229] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0230] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0231] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server or a network device, etc., specifically a processor in a computer device) to perform all or part of the steps of the above-mentioned methods of each embodiment of the present application. Among them, the aforementioned storage medium may include: U disk, mobile hard disk, magnetic disk, optical disk, read-only memory (Read-Only Memory, abbreviated: ROM) or random access memory (Random Access Memory, abbreviated: RAM) and other media that can store program codes.
[0232] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.
Claims
1. A method for generating a virtual image, It is characterized in that Applied to electronic equipment, the method comprises: The electronic device acquires a first image including a face; The electronic device acquires a first 3D face model; The electronic device acquires a first stylization parameter related to the first image selected by a user; The electronic device generates and displays a first virtual image based on the first image, the first 3D face model, and the first stylization parameter; the first stylization parameter indicates the influence ratio of the first image and the first 3D face model on the first virtual image.
2. The method according to claim 1, It is characterized in that Before the electronic device acquires the first stylization parameter related to the first image selected by the user, the electronic device further includes: The electronic device generates and displays an initial virtual image based on the first image, the first 3D face model, and default stylization parameters; the default stylization parameters indicate the influence ratio of the first image and the first 3D face model on the initial virtual image.
3. The method according to claim 2, It is characterized in that The electronic device generates and displays an initial virtual image according to the first image, the first 3D face model, and default stylization parameters, including: The electronic device generates a real face model according to the first image and the first 3D face model; The electronic device generates a first face model according to the real face model, the first 3D face model, and the default stylization parameters; The electronic device generates the initial virtual image according to the first face model and displays a first user interface, in which the initial virtual image is displayed.
4. The method according to any one of claims 1 to 3, It is characterized in that The electronic device acquires a first image including a face, including: The electronic device displays a second user interface, the second user interface includes a first preview area and a first control, the first preview area displays an image containing a face from a camera of the electronic device; The electronic device receives an operation applied to the first control and saves the image displayed in the first preview area as the first image.
5. The method according to any one of claims 1 to 3, It is characterized in that The electronic device acquires a first image including a face, including: The electronic device displays a third user interface; The electronic device receives an operation performed on the third user interface and displays a fourth user interface, wherein the fourth user interface displays one or more images, and the one or more images include a thumbnail of the first image; The electronic device receives an operation performed on a thumbnail of the first image and acquires the first image.
6. The method according to any one of claims 1 to 5, It is characterized in that Before the electronic device acquires the first 3D face model, the method further includes: The electronic device displays a fifth user interface, wherein the fifth user interface displays indication information of one or more 3D face models, wherein the one or more 3D face models include the first 3D face model; The electronic device receives instruction information acting on the first 3D face model.
7. The method according to any one of claims 3 to 6, It is characterized in that The electronic device generates and displays a first virtual image according to the first image, the first 3D face model, and the first stylized parameter, including: The electronic device generates a second face model according to the real face model, the first 3D face model, and the first stylization parameter; The electronic device generates the first virtual image according to the second face model and displays the first virtual image in the first user interface.
8. The method according to any one of claims 3 to 7, It is characterized in that The electronic device acquires a first stylization parameter related to the first image selected by a user, including: The electronic device receives a first operation acting on the first user interface, where the first operation is used to adjust a stylization parameter; In response to the first operation, the first stylization parameter selected by the user is determined.
9. The method according to claim 7, It is characterized in that The first user interface includes a slide bar control, and the first operation is an operation of sliding the slide bar control to adjust a stylized parameter.
10. The method according to any one of claims 3 to 9, It is characterized in that The method further comprises: The electronic device receives the adjustment operation on the first user interface for the first virtual image, adjusts the facial features of the first virtual image, and obtains an updated first virtual image; The electronic device displays the updated first virtual image on the first user interface.
11. The method according to any one of claims 7 to 10, It is characterized in that The electronic device generates the first virtual image according to the second face model, including: The electronic device renders the second face model to obtain the first virtual image.
12. The method according to any one of claims 7 to 10, It is characterized in that The electronic device generates the first virtual image according to the second face model, including: The electronic device obtains a default 3D material, and renders the second face model based on the default 3D material to obtain the first virtual image, wherein the default 3D material includes one or more of body features and clothing features of the 3D character model.
13. An electronic device, It is characterized in that The electronic device includes a first processing unit, which is used to: obtain a first image including a face; obtain a first 3D face model; obtain a first stylization parameter related to the first image selected by a user; generate and display a first virtual image based on the first image, the first 3D face model, and the first stylization parameter; the first stylization parameter indicates the proportion of influence of the first image and the first 3D face model on the first virtual image.
14. An electronic device, It is characterized in that include: A memory and one or more processors; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to enable the electronic device to execute the method as described in any one of claims 1-9.
15. A chip system, It is characterized in that The chip system includes at least one processor, a memory and an interface circuit. The memory, the interface circuit and the at least one processor are interconnected via lines. Instructions are stored in the at least one memory. When the instructions are executed by the processor, the method described in any one of claims 1 to 12 is implemented.
16. A computer-readable storage medium comprising instructions, It is characterized in that When the instructions are executed on an electronic device, the electronic device executes the method as claimed in any one of claims 1 to 12.
17. A computer program product, It is characterized in that The computer program product comprises computer instructions, and when the computer instructions are run on an electronic device, the electronic device executes the method as claimed in any one of claims 1 to 12.
Citation Information
Cited By
Digital human beautifying method, electronic equipment and storage medium
CN120526017A
