Method, apparatus, device and storage medium for image optimization rendering

Abstract

This application relates to the field of computer technology, and particularly to the field of artificial intelligence technology. It discloses a method, device, equipment and storage medium for image optimization and rendering to solve the problem of poor image rendering effect. The method includes: determining the actual eye region from the acquired image to be processed, and based on each first eye key point included in the actual eye region, adjusting the shape of the standard eye region including the virtual makeup effect to obtain a simulated eye region including each second eye key point, so that each first eye key point corresponds to each second eye key point one by one; based on the eye state corresponding to the actual eye region, extracting the corresponding rendering materials from the simulated eye region, and in the image to be processed, performing fusion rendering processing on each second eye key point included in the rendering materials and the corresponding first eye key point respectively to obtain the target image. This improves the fitting degree of the simulated eye region and the overall rendering effect.

Description

Background Art

[0002] With the popularization of social platforms such as live streaming platforms and content sharing platforms, more and more people like to share their daily lives in the form of photos or videos on various social platforms. In order to optimize the image of people in photos or videos and enhance the beauty of photos or videos, social platforms have launched virtual makeup functions, especially the simulated eye makeup function for the eye area.

[0003] The following three simulated eye makeup schemes are provided in the related art. Respectively: Scheme 1, by constructing complex triangular patches based on the key points of the human eye to locate the position and shape of the eye area, and then tracking the position and shape of the eye area through methods such as template matching, and pasting the eyeshadow material on the template onto the eye area; Scheme 2, by using the Free-Form Deformation (FFD) algorithm to render the eyeshadow material on the eye area; Scheme 3, pasting the designed three-dimensional (3D) material or 3D sticker onto the eye area.

[0004] However, when performing image rendering using the above schemes, the following problems will occur:

[0005] When there is a large deformation in the eye area due to the movement of the eyebrows or actions such as opening and closing the eyes, the method of constructing triangular patches is difficult to simulate the accurate shape of the eye area, which will give people an unrealistic visual experience, and the robustness of this scheme is relatively low; secondly, when blinking, the rendering range of the eye area will also change, but Scheme 1 cannot accurately locate the new rendering range, resulting in the rendering range on the simulated eye makeup rendering diagram being inconsistent with the rendering range of the actual eye makeup.

[0006] The FFD algorithm is difficult to accurately complete the deformation according to the designer's intention, and the shape, size, and position of the deformed eye area are often not precise enough. Therefore, the makeup simulation effect of the simulated eye makeup rendering diagram output by Scheme 2 is also poor.

[0007] The 3D texture mapping scheme in Scheme 3 strongly depends on the accuracy and stability of the key point model output points. In side face and other angles, it is easy to have situations where the texture flies out and the fitting is not perfect; moreover, the 3D texture mapping scheme requires a large amount of computing power, and the design difficulty of 3D eyeshadow materials is much higher than that of 2D eyeshadow materials, which also brings great resistance in terms of design. Summary of the Invention

[0008] Embodiments of the present application provide a method, device, equipment, and storage medium for image optimization rendering to solve the problem of poor image rendering effect.

[0009] In the first aspect, the method for image optimization rendering provided by the embodiments of the present application includes:

[0010] Obtain an image to be processed and determine the actual eye region in the image to be processed;

[0011] Obtain a preset standard eye region including virtual makeup effects, where the standard eye region is determined based on a standard face image, and the standard face image is obtained by averaging various types of face templates and is a face image in a frontal state;

[0012] Based on each first eye key point included in the actual eye region, adjust the shape of the standard eye region to obtain a simulated eye region including each second eye key point, where each of the first eye key points and each of the second eye key points corresponds one by one;

[0013] Based on the eye state corresponding to the actual eye region, extract corresponding rendering materials from the simulated eye region, and in the image to be processed, perform a fusion rendering process on each second eye key point included in the rendering materials and the corresponding first eye key point respectively to obtain a target image.

[0014] In a second aspect, an embodiment of the present application further provides an image optimization and rendering device, including:

[0015] An acquisition unit, configured to obtain an image to be processed and determine the actual eye region in the image to be processed;

[0016] Obtain a preset standard eye region including virtual makeup effects, where the standard eye region is determined based on a standard face image, and the standard face image is obtained by averaging various types of face templates and is a face image in a frontal state;

[0017] A processing unit, configured to adjust the shape of the standard eye region based on each first eye key point included in the actual eye region to obtain a simulated eye region including each second eye key point, where each of the first eye key points and each of the second eye key points corresponds one by one;

[0018] A rendering unit, configured to extract corresponding rendering materials from the simulated eye region based on the eye state corresponding to the actual eye region, and in the image to be processed, perform a fusion rendering process on each second eye key point included in the rendering materials and the corresponding first eye key point respectively to obtain a target image.

[0019] Optionally, the rendering unit determines the eye state corresponding to the actual eye region in the following manner:

[0020] Obtain the eyelid distance between the upper eyelid and the lower eyelid in the actual eye region;

[0021] If the eyelid distance exceeds a set second distance threshold, it is determined that the corresponding state of the actual eye region is the open-eye state;

[0022] Otherwise, it is determined that the corresponding state of the actual eye region is the closed-eye state.

[0023] Optionally, the rendering unit is configured to:

[0024] Obtain the first distance of each pixel point in the simulated eye region, and filter to obtain a plurality of pixel points whose first distance does not exceed a set first distance threshold;

[0025] Determine the region of the plurality of pixel points in the simulated eye region as the inner contour region.

[0026] Optionally, the rendering unit is configured to:

[0027] In the image to be processed, perform a fusion rendering process on each second eye key point included in the rendering material with the corresponding first eye key point. Among them, each time a fusion rendering process is performed for one second eye key point, it is determined based on the rendering material color of the one second eye key point, the pixel point information of the corresponding one first eye key point, and the skin color fusion coefficient. The skin color fusion coefficient is determined based on the eye region skin color of the actual eye region and the corresponding rendering material color.

[0028] Optionally, after obtaining the target image, the rendering unit is further configured to:

[0029] Randomly select at least one eye pixel point from the eye region of the target image;

[0030] Perform the following operations on the at least one eye pixel point respectively to obtain a target image including a fine flash effect:

[0031] Randomly select a texture pixel point on a preset texture image including a fine flash effect;

[0032] Update the pixel value of the one eye pixel point based on the pixel value of the one eye pixel point and the pixel value of the one texture pixel point.

[0033] Optionally, the obtaining unit obtains the image to be processed in any of the following ways:

[0034] In response to an image acquisition instruction triggered by a target object, obtain a frame of image collected from an image acquisition device, and use the frame of image as the image to be processed;

[0035] In response to an image acquisition instruction triggered by the target object, obtain the acquired video stream from an image acquisition device, and select any frame of the image from the video stream as the image to be processed.

[0036] In a third aspect, an embodiment of the present application further provides a computer device, including a processor and a memory. Among them, the memory stores program code, and when the program code is executed by the processor, the processor is caused to execute the steps of any of the above image optimization and rendering methods.

[0037] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, which includes program code. When the program product runs on a computer device, the program code is used to cause the computer device to execute the steps of any of the above image optimization and rendering methods.

[0038] The beneficial effects of the present application are as follows:

[0039] An embodiment of the present application provides an image optimization and rendering method, device, equipment, and storage medium. The method includes: determining an actual eye region from the acquired image to be processed, and based on each first eye key point included in the actual eye region, adjusting the shape of a standard eye region including a virtual makeup effect to obtain a simulated eye region including each second eye key point, and making each first eye key point correspond to each second eye key point one by one; based on the eye state corresponding to the actual eye region, extracting corresponding rendering materials from the simulated eye region, and in the image to be processed, performing a fusion rendering process on each second eye key point included in the rendering materials and the corresponding first eye key point respectively to obtain a target image. The image optimization and rendering solution provided by the embodiment of the present application adopts a curve fitting modeling solution, which can better simulate the morphological changes of the eyes, making the constructed simulated eye region more realistic; and the curve fitting modeling solution, combined with an eye positioning solution based on eye corner key points, can obtain a more stable rendering range, which is beneficial to improving the overall rendering effect.

[0040] Other features and advantages of the present application will be described in the following specification, and, in part, will become apparent from the specification, or will be understood by implementing the present application. The objectives and other advantages of the present application can be achieved and obtained by the structures specifically pointed out in the written specification, claims, and drawings. Description of the Drawings

[0041] The drawings described herein are used to provide a further understanding of the present application, and constitute a part of the present application. The illustrative embodiments and descriptions thereof of the present application are used to explain the present application, and do not constitute an improper limitation to the present application. In the drawings:

[0042] Figure 1aAn optional schematic diagram of an application scenario in an embodiment of this application;

[0043] Figure 1b Schematic diagram of the application operation interface in an embodiment of this application;

[0044] Figure 2a Schematic flowchart of the image optimization rendering method provided by an embodiment of this application;

[0045] Figure 2b Schematic diagram of the actual eye region provided by an embodiment of this application;

[0046] Figure 2c Schematic diagram of the standard eye region provided by an embodiment of this application;

[0047] Figure 3a Schematic flowchart of the process of adjusting the shape of the standard eye region of the left eye provided by an embodiment of this application;

[0048] Figure 3b Schematic diagram of the deviation information between the actual eye region and the standard eye region provided by an embodiment of this application;

[0049] Figure 3c Schematic diagram of the comparison before and after rotation of the standard eye region provided by an embodiment of this application;

[0050] Figure 3d Schematic diagram of the comparison before and after scaling of the rotated standard eye region provided by an embodiment of this application;

[0051] Figure 3e Schematic diagram of the simulated eye region provided by an embodiment of this application;

[0052] Figure 4a Schematic diagram of the actual eye region in the open-eye state provided by an embodiment of this application;

[0053] Figure 4b Schematic diagram of the actual eye region in the closed-eye state provided by an embodiment of this application;

[0054] Figure 4c Schematic diagram of the simulated eye region of the left eye including a rectangular coordinate system provided by an embodiment of this application;

[0055] Figure 4d Schematic diagram of the target image including a simulated matte eye makeup provided by an embodiment of this application;

[0056] Figure 4e Schematic diagram of the target image including a simulated fine-shimmer eye makeup provided by an embodiment of this application;

[0057] Figure 5Schematic diagram of the virtual makeup process in the live broadcast scenario provided by the embodiments of the present application;

[0058] Figure 6 Schematic diagram of the structure of an image optimization and rendering device provided by the embodiments of the present application;

[0059] Figure 7 Schematic diagram of the composition structure of a computer device provided in the embodiments of the present application;

[0060] Figure 8 Schematic diagram of the structure of a computing device in the embodiments of the present application. Detailed implementation manners

[0061] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the technical solutions of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the technical solutions of the present application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments recorded in this application document without creative efforts shall fall within the scope of protection of the technical solutions of the present application.

[0062] The following explains some terms in the embodiments of the present application to facilitate understanding by those skilled in the art.

[0063] The embodiments of the present application relate to the field of artificial intelligence (AI), and are designed based on machine learning (ML) and computer vision (CV) technologies.

[0064] Artificial intelligence uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, a theory, method, technology, and application system that perceives the environment, acquires knowledge, and uses knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science. It studies the design principles and implementation methods of various machines, attempts to understand the essence of intelligence, and produces a new intelligent machine that can react in a way similar to human intelligence, enabling the machine to have the functions of perception, reasoning, and decision-making.

[0065] Artificial intelligence is an interdisciplinary subject that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technologies, operation interaction systems, mechatronics, and other technologies. The software technologies of artificial intelligence mainly include several major directions such as computer vision technology, speech processing technology, natural language processing technology, machine learning / deep learning, autonomous driving, and intelligent transportation. With the development and progress of artificial intelligence, it has been able to conduct research and applications in multiple fields. For example, common fields include smart home, intelligent customer service, virtual assistants, smart speakers, intelligent marketing, smart wearable devices, driverless, autonomous driving, drones, robots, intelligent healthcare, vehicle networking, autonomous driving, and intelligent transportation. It is believed that with the further development of future technologies, artificial intelligence will be applied in more fields and play an increasingly important role. The solution provided in the embodiments of this application involves technologies such as deep learning and augmented reality in artificial intelligence, which will be further described through the following embodiments.

[0066] Machine learning is an interdisciplinary subject that involves multiple disciplines such as probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate human learning behaviors to acquire new knowledge or skills, reorganize existing knowledge structures, and continuously improve their own performance.

[0067] Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications cover all fields of artificial intelligence. The core of machine learning is deep learning, which is a technology to achieve machine learning. Machine learning usually includes technologies such as deep learning, reinforcement learning, transfer learning, inductive learning, artificial neural networks, and formal teaching learning. Deep learning includes technologies such as Convolutional Neural Networks (CNNs), deep belief networks, recurrent neural networks, autoencoders, and generative adversarial networks.

[0068] Computer vision is a comprehensive discipline that combines multiple disciplines such as computer science, signal processing, physics, applied mathematics, statistics, and neurophysiology. It is also an important and challenging research direction in the scientific field. Computer vision is a discipline that studies how to enable machines to "see". More specifically, this discipline refers to using various imaging systems such as cameras and computers to replace the visual organs of humans, perform machine vision processing such as target recognition, tracking, and measurement on the targets, and through further graphics processing, process the captured images into images that are more suitable for human eyes to observe or transmit to instruments for detection.

[0069] As a scientific discipline, computer vision attempts to enable computers to observe and understand the world through visual organs like humans by studying relevant theories and technologies, and to establish an artificial intelligence system capable of obtaining information from images or multi-dimensional data. Computer vision technologies generally include image processing, image recognition, image semantic understanding, image retrieval, optical character recognition (OCR), video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping, autonomous driving, intelligent transportation, etc. In addition, computer vision technologies also include common biometric recognition technologies such as face recognition and fingerprint recognition.

[0070] Augmented Reality (AR) technology simulates and emulates physical information (such as visual information, sound information, taste information, tactile information, etc.) that is difficult to experience within a certain time and space range in the real world through scientific and technological means such as computers, and then overlays the virtual information onto the real world, achieving the simultaneous display of two different types of information, the environment of the real world and the objects of the virtual world, to the user in the same picture or space, resulting in a sensory experience that transcends reality.

[0071] The following briefly introduces the design concept of the embodiments of this application:

[0072] With the popularization of social platforms such as live streaming platforms and content sharing platforms, more and more people like to share their daily lives in the form of photos or videos on various social platforms. To optimize the image of people in photos or videos and enhance the aesthetic feeling of photos or videos, social platforms have launched virtual makeup functions, especially the simulated eye makeup function for the eye area.

[0073] The following three simulated eye makeup schemes are provided in the related technologies. Respectively: Scheme 1, locate the position and shape of the eye area by constructing complex triangular patches based on human eye key points, and then track the position and shape of the eye area through methods such as template matching, and paste the eyeshadow material on the template onto the eye area; Scheme 2, render the eyeshadow material on the eye area through the FFD algorithm; Scheme 3, paste the designed 3D material or 3D sticker onto the eye area.

[0074] However, when performing image rendering using the above schemes, the following problems will occur:

[0075] When large deformations occur in the eye region due to eyebrow movement or actions such as opening and closing the eyes, the method of constructing triangular patches is difficult to accurately simulate the shape of the eye region, which will give people an unrealistic visual experience, and the robustness of this solution is relatively low. Secondly, when blinking, the rendering range of the eye region also changes, but Solution 1 cannot accurately locate the new rendering range, resulting in inconsistent rendering ranges between the simulated eye makeup rendering and the actual eye makeup rendering.

[0076] The FFD algorithm is difficult to accurately complete the deformation according to the designer's intention, and the shape, size, and position of the deformed eye region are often not precise enough. Therefore, the makeup simulation effect of the simulated eye makeup rendering output by Solution 2 is also poor.

[0077] The 3D texture mapping solution in Solution 3 strongly depends on the accuracy and stability of the key point model output points. In side face and other angles, it is easy to have situations such as the texture flying out and imperfect fitting. Moreover, the 3D texture mapping solution requires a large amount of computing power, and the design difficulty of 3D eyeshadow materials is much higher than that of 2D eyeshadow materials, which also brings great resistance in terms of design.

[0078] In view of this, the embodiments of the present application propose a method, device, equipment, and storage medium for image optimization rendering. In this method, first, a key point recognition model based on deep learning is used to determine the actual eye region in the image to be processed; then, based on augmented reality technology, the rendering materials on the standard eye region are fused and rendered with the actual eye region to obtain a target image containing virtual makeup effects, so as to solve the problem of poor image rendering effects in existing simulated eye makeup solutions.

[0079] Specifically, the method includes: obtaining the image to be processed and determining the actual eye region in the image to be processed; then obtaining a standard eye region containing virtual makeup effects, where the standard eye region is obtained by averaging various types of eye templates and is the eye region in the frontal view state; based on the eye state corresponding to the actual eye region, extracting the corresponding rendering materials from the simulated eye region, and in the image to be processed, respectively fusing and rendering each second eye key point included in the rendering materials with the corresponding first eye key point to obtain the target image.

[0080] The following describes the preferred embodiments of the present application with reference to the accompanying drawings of the specification. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present application, and are not used to limit the present application. And without conflict, the embodiments in the present application and the features in the embodiments can be combined with each other.

[0081] This application can be applied not only to scenarios with image beautification requirements such as live streaming platforms and content sharing platforms, but also to daily scenarios such as video calls, virtual makeup try-on in offline counters or online e-commerce platforms. For example, Figure 1a As shown, it is a schematic diagram of the application scenario of an embodiment of this application. The application scenario diagram includes two physical terminal devices 110 and a server 130.

[0082] Users can log in to the application operation interface 120 of the social platform through the physical terminal device 110. In the embodiment of this application, the physical terminal device 110 is an electronic device used by the user, and this electronic device can be a computer device such as a personal computer, mobile phone, tablet computer, notebook, e-book reader, smart home, etc.

[0083] The interface schematic diagram of the application operation interface 120 is as shown in Figure 1b As shown, the interface includes a shooting button, a viewfinder, a function bar, a quick entry to the gallery, and a button to flip the camera direction. When shooting, the user can adjust the focal length, the target focusing object, and the picture brightness by touching the viewfinder interface, click the button to flip the camera direction to adjust the main camera currently used for shooting, and can also implement functions such as shooting mode switching, adding filters, deleting filters, adding stickers, deleting stickers, etc. through the corresponding functions in the function bar. If in the photo shooting mode, the user clicks the shooting button to get a corresponding photo, and the user long-presses the shooting button to turn on the continuous shooting function to get multiple consecutive photos; if in the video shooting mode, the user can trigger video shooting and pause video shooting in the following two ways. One is to click the shooting button to trigger video shooting, and click the shooting button again to pause video shooting; the other is to long-press the shooting button for more than the set time to trigger video shooting, and pause video shooting when lifting the finger. By clicking the quick entry to the gallery, it jumps to the gallery interface to view the captured images and videos, and can also edit and beautify the photos and videos.

[0084] Each physical terminal device 110 communicates with the server 130 through a communication network. In an alternative embodiment, the communication network is a wired network or a wireless network. Therefore, each physical terminal device 110 can directly or indirectly establish a communication connection with the server 130 through a wired network or a wireless network, and this application does not make any restrictions here. The server 130 can be an independent physical server, or a server 130 cluster or distributed system composed of multiple physical servers, or a cloud server 130 that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, Content Delivery Network (CDN), big data, and artificial intelligence platforms. This application does not make any restrictions here.

[0085] Among them, the simulated eye makeup system in the embodiments of the present application is deployed on the server 130. The server 130 receives the face image or the video containing the face image sent by the physical terminal device 110, and sequentially performs the steps of constructing an eye model, deforming and positioning the standard eye model, and fusion rendering on the face image or the video containing the face image, and returns the rendered image containing the simulated eye makeup effect to the physical terminal device 110, so as to achieve the purpose of optimizing the image of the person in the photo or video.

[0086] Refer to Figure 2a the flowchart shown to introduce the image optimization and rendering method provided by the embodiments of the present application.

[0087] S201: The physical terminal device obtains the image to be processed and determines the actual eye region in the image to be processed.

[0088] The image to be processed includes one or more face images, and each face image includes at least an eye region image. In addition to the face image, the image to be processed may also include background images such as flowers, plants, and room environments. When performing step 201, the embodiments of the present application provide the following two ways to obtain the image to be processed:

[0089] One way is to obtain a frame of the image collected from the image collection device in response to the image collection instruction triggered by the target object, and use this frame of the image as the image to be processed;

[0090] Another way is to obtain the video stream collected from the image collection device in response to the image collection instruction triggered by the target object, and select any frame of the image from the video stream as the image to be processed.

[0091] For example, the smart phone presents a camera interface capable of realizing the camera function to the user in response to the operation of the user clicking the camera button of the smart phone; then, the smart phone calls the camera to take a photo in response to the operation of the user clicking the shooting button on the camera interface, and uses this photo as the image to be processed.

[0092] For another example, the image beautification application presents a gallery interface to the user in response to the operation of the user clicking the gallery button. The gallery interface contains the images and video streams that have been taken; then, the image beautification application uses the image selected by the user as the image to be processed in response to the click operation of the user, or selects any frame of the image from the video stream selected by the user as the image to be processed.

[0093] After obtaining the image to be processed, input the image to be processed into a preset eye region recognition model. After steps such as feature extraction and normalization processing inside the model, output such as Figure 2bThe actual eye region on the image to be processed, and each first eye key point that constitutes the actual eye region.

[0094] S202: The physical terminal device obtains a preset standard eye region including virtual makeup effects. The standard eye region is determined based on a standard face image, and the standard face image is obtained by averaging various types of face templates, which is a face image in a frontal state.

[0095] When performing step 202, it includes the following two steps: The first step is to draw rendering materials (rendering materials include but are not limited to, simulated eyeshadow, simulated eyeliner, simulated eyelashes) in the eye region of the standard face image. The second step is to model the eye region of the standard face image using polynomial modeling to obtain the Figure 2c standard eye region as shown.

[0096] The so-called standard face image refers to a standard face image obtained by averaging various types of face templates (such as round face, square face, long face, pointed face, etc.). Since the head pose of the face image in the standard face image is in a frontal state and the positions of the facial features do not shift, therefore, the standard eye region obtained based on this standard face image has eyes in the eye region that do not shift and are always in a frontal state looking straight ahead.

[0097] As Figure 2c shown, the standard eye region includes the standard eye region of the left eye and the standard eye region of the right eye. In order to better fit the shape of the standard eye region, the standard eye region of each eye uses four parabolas respectively to fit the upper eyelid and lower eyelid of the corresponding eye.

[0098] Specifically, taking the left eye as an example, describe the process of constructing the standard eye region of this eye. Based on the third eye key point (Key Point, KP) KP1 representing the left eye corner and the third eye key point KP5 representing the left eye tail, establish the X1 axis. Draw a Y1 axis perpendicular to the X1 axis through the third eye key point KP3 on the upper eyelid, and take the intersection point of the X1 axis and the Y1 axis as the origin O1 of the plane rectangular coordinate system x1O1y1. Taking KP3 as the boundary, divide the upper eyelid into two parabolas, one is parabola 2 including KP1 - KP3, and the other is parabola 1 including KP3 - KP5.

[0099] Similarly, draw a Y2 axis perpendicular to the X1 axis through the third eye key point KP7 on the lower eyelid, and take the intersection point of the X1 axis and the Y2 axis as the origin O2 of the plane rectangular coordinate system x1O2y2. Taking KP7 as the boundary, divide the lower eyelid into two parabolas, one is parabola 3 including KP5 - KP7, and the other is parabola 4 including KP1, KP7 and KP8.

[0100] In addition to using the four-segment parabola method to locate the eye contour on the standard face image and obtain the standard eye region, other curve models such as polynomial fitting can also be used to locate the eye contour on the standard face image and obtain the standard eye region.

[0101] Since the corresponding rendering materials have been drawn on the eye region of the standard face image before constructing the standard eye region, the standard eye region obtained based on the standard face image will also contain the rendering materials that can form the virtual makeup effect. After that, by performing step 203 to adjust the shape of the standard eye region, a simulated eye region containing each second eye key point is obtained. At this time, the simulated eye region also contains the rendering materials that can form the virtual makeup effect. Therefore, by performing step 204, the corresponding rendering materials can be extracted from the simulated eye region, and each second eye key point contained in the rendering materials is respectively subjected to fusion rendering processing with the corresponding first eye key point on the actual eye region to obtain the target image.

[0102] S203: The physical terminal device adjusts the shape of the standard eye region based on each first eye key point included in the actual eye region to obtain a simulated eye region containing each second eye key point, where each first eye key point and each second eye key point correspond one by one.

[0103] When performing step 203, the mapping relationship between each first eye key point in the actual eye region and each third eye key point in the standard eye region is determined according to the deformation parameters of the actual eye region, and based on this mapping relationship, the shape of the standard eye region is adjusted to obtain a simulated eye region containing each second eye key point.

[0104] Specifically, taking the left eye as an example, the process of adjusting the shape of the standard eye region of this eye is as Figure 3a shown.

[0105] S2031: The physical terminal device determines the deviation information between the actual eye region and the standard eye region based on each first eye key point included in the actual eye region, and adjusts the offset angle of the standard eye region based on the deviation information.

[0106] For the convenience of description, taking the left eye as an example, the process of determining the deviation information between the actual eye region of the left eye and the standard eye region is introduced.

[0107] In view of the fact that when the existing eye region recognition model recognizes the eye key points, the recognition of the eye key points representing the eye corners is the most stable. Therefore, in the embodiments of the present application, it is defaulted that the position of the first eye key point KP1' representing the left eye corner in the actual eye region is the same as the position of the third eye key point KP1 representing the left eye corner in the standard eye region.

[0108] Draw two straight lines l1 and l2 as shown in Figure 3b ; among them, the straight line l1 represents the eye deviation direction of the actual eye region, and the straight line l2 represents the eye deviation direction of the standard eye region. Since the eyes in the standard eye region are always in the frontal state of looking straight ahead, therefore, Figure 3b the straight line l2 in is a straight line that horizontally passes through the first eye key point KP1'.

[0109] According to Figure 3b it can be known that there is an offset angle β between the straight line l1 and the straight line l2, and this offset angle β is exactly the offset angle that the standard eye region needs to rotate. In order to solve the deviation information (sinβ, cosβ), draw a straight line l3 perpendicular to l2 through the first eye key point KP5' representing the left eye tail, forming a right triangle KP1'KP5'KP9'. Then, according to the relationship between the angles and side lengths in the right triangle KP1'KP5'KP9', the deviation information (sinβ, cosβ) can be solved. Specifically, take the coordinate positions of KP1' and KP5' as the deformation parameters of the actual eye region, and substitute them into formula 1 to solve for the deviation information (sinβ, cosβ).

[0110]

[0111]

[0112] Among them, KP1'.x represents the abscissa of KP1', KP1'.y represents the ordinate of KP1', while KP5'.x represents the abscissa of KP5', and KP5'.y represents the ordinate of KP5'.

[0113] As shown in formula 2, based on the deviation information (sinβ, cosβ) and the coordinate positions of each third eye key point in the standard eye region, determine the coordinate positions of the corresponding rotated eye key points, and based on the coordinate positions of the rotated eye key points, generate a rotation schematic diagram as shown in Figure 3c .

[0114]

[0115] Among them, (x, y) represents the horizontal and vertical coordinates of a third eye key point, (x', y') represents the coordinate position of the corresponding rotated eye key point, and (sinβ, cosβ) refers to the deviation information between the actual eye region and the standard eye region.

[0116] S2032: The physical terminal device adjusts the eye size of the rotated standard eye region according to the eye size of the actual eye region to obtain a scaled and adjusted standard eye region.

[0117] After performing the rotation operation on the standard eye region, the length ratio value between the second eye length of the rotated standard eye region and the first eye length of the actual eye region is determined as the scaling ratio value of the rotated standard eye region; then, based on the scaling ratio value, the eye size of the rotated standard eye region is adjusted to obtain a scaled and adjusted standard eye region.

[0118] Specifically, taking the left eye as an example, based on the coordinate positions of the first eye key points KP1' and KP5', the first eye length I is calculated, and based on the coordinate positions of the third eye key points KP1 and KP5 of the rotated standard eye region, the second eye length L is calculated. L / I is used as the scaling ratio value to adjust the X-axis and Y-axis of the rotated standard eye region respectively, to obtain Figure 3d the scaled and adjusted standard eye region as shown.

[0119] S2033: The physical terminal device updates the contour shape of the scaled and adjusted standard eye region to obtain a simulated eye region.

[0120] The scaled and adjusted standard eye region still consists of four parabolas. Among them, parabola 1 consists of three third eye key points KP3 - KP5, parabola 2 consists of three third eye key points KP1 - KP3, parabola 3 consists of three third eye key points KP5 - KP7, and parabola 4 consists of three third eye key points KP1, KP7, and KP8.

[0121] Therefore, based on the coordinate positions of the third eye key points KP3 - KP5 of the scaled and adjusted standard eye region, the values of a1 and c1 of parabola 1 are recalculated as shown in formula 3 to update the curve equation of parabola 1. The update process of the curve equations of the other three parabolas is the same as that of parabola 1, so it will not be elaborated here.

[0122]

[0123] c = KP3.y - a*(KP3.x) 2 Formula 3;

[0124] Among them, a is the quadratic coefficient, c is the constant term, KP3.x is the abscissa of KP3, KP3.y is the ordinate of KP3, KP5.x is the abscissa of KP5, and KP5.y is the ordinate of KP5.

[0125] After performing the above steps 2031 to 2033, a simulated eye region as Figure 3e shown is obtained. The eye offset direction, contour shape, and eye size of the simulated eye region are more in line with the actual eye region, facilitating subsequent rendering and coloring processing to obtain the target image.

[0126] S204: The physical terminal device extracts corresponding rendering materials from the simulated eye region based on the eye state corresponding to the actual eye region, and in the image to be processed, performs fusion rendering processing on each second eye key point included in the rendering materials with the corresponding first eye key point respectively to obtain the target image.

[0127] Referring to Figure 3e the schematic diagram shown, rendering materials have been drawn for the entire eye region of the simulated eye region. When the user is in the open-eye state, there is no need to draw rendering materials on the eyes of the actual eye region, and only need to draw rendering materials around the eyes of the actual eye region (including any one or combination of the upper eyelid and the lower eyelid). When the user is in the closed-eye state, rendering materials need to be drawn for the entire eye region of the actual eye region. It can be seen from this that when the user is in different eye states, the rendering materials of the corresponding simulated eye region are also different. Therefore, before performing the fusion rendering processing, it is necessary to first determine the eye state corresponding to the actual eye region, and then based on the eye state, extract the corresponding rendering materials from the simulated eye region.

[0128] Referring to Figure 2b the schematic diagram shown, in the embodiment of the present application, the eyelid distance between the upper eyelid and the lower eyelid in the actual eye region can be determined according to the coordinate positions of the first eye key points KP3' and KP7' of the actual eye region. If the eyelid distance exceeds the set second distance threshold, it is determined that the actual eye region corresponds to the open-eye state; otherwise, it is determined that the actual eye region corresponds to the closed-eye state.

[0129] Figure 4a The eyelid distance in the actual eye region shown exceeds the second distance threshold and is in the open-eye state; Figure 4b The eyelid distance in the actual eye region shown is less than the second distance threshold and is in the closed-eye state.

[0130] If the actual eye region corresponds to the open-eye state, based on the first distances of each pixel point in the simulated eye region and a set first distance threshold, determine the internal contour region in the simulated eye region, remove the internal contour region from the simulated eye region, and extract the region corresponding to the open-eye state from the simulated eye region as the rendering material;

[0131] If the actual eye region corresponds to the closed-eye state, extract the region corresponding to the closed-eye state from the simulated eye region as the rendering material.

[0132] Next, taking the left eye as an example, introduce the process of determining the internal contour region in the simulated eye region of this eye.

[0133] As Figure 4c shown, establish a rectangular coordinate system x1"O1"y1" and a rectangular coordinate system x1"O2"y2" on the simulated eye region of the left eye respectively. Among them, the X1" axis is established based on the second eye key point KP1" representing the left eye corner and the second eye key point KP5" representing the left eye tail of the simulated eye region. Draw a Y1" axis perpendicular to the X1" axis through the second eye key point KP3" on the upper eyelid, and take the intersection point of the X1" axis and the Y1" axis as the origin O1" of the rectangular coordinate system x1"O1"y1". Taking KP3" as the boundary, divide the upper eyelid into two parabolas. One is parabola 2" containing KP1"~KP3", and the other is parabola 1" containing KP3"~KP5".

[0134] Similarly, draw a Y2" axis perpendicular to the X1" axis through the third eye key point KP7" on the lower eyelid, and take the intersection point of the X1" axis and the Y2" axis as the origin O2" of the rectangular coordinate system x1"O2"y2". Taking KP7" as the boundary, divide the lower eyelid into two parabolas. One is parabola 3" containing KP5"~KP7", and the other is parabola 4" containing KP1", KP7" and KP8".

[0135] To facilitate the determination of which pixel points in the simulated eye region are inside the internal contour region, perform coordinate transformation on the rectangular coordinate systems x1"O1"y1" and x1"O2"y2" to obtain the corresponding polar coordinate systems (r1, θ1), (r2, θ2). Among them, r1 represents the first distance between the pixel point m in the rectangular coordinate system x1"O1"y1" and the pole, and θ1 represents the angle between the pixel point m in the rectangular coordinate system x1"O1"y1" and the polar coordinate system. Similarly, r2 represents the first distance between the pixel point n in the rectangular coordinate system x1"O2"y2" and the pole, and θ2 represents the angle between the pixel point n in the rectangular coordinate system x1"O2"y2" and the polar coordinate system.

[0136] In the embodiments of the present application, the first distance of each pixel point can be calculated based on the polar coordinate system (r1, θ1) converted from the rectangular coordinate system x1"O1"y1", or the first distance of each pixel point can be calculated based on the polar coordinate system (r2, θ2) converted from the rectangular coordinate system x1"O2"y2", and no limitation is made thereto.

[0137] Substitute the coordinate position (x", y") of the pixel point in the rectangular coordinate system and the included angle θ between the pixel point and the polar coordinate system into Formula 4 to calculate the corresponding first distance r.

[0138] y" = r * sinθ;

[0139] x" = r * cosθ Formula 4;

[0140] According to Figure 4c As can be seen from the shown schematic diagram, the first distance threshold represents the maximum distance R between the pole and the parabola corresponding to the pixel point when the included angle between the pixel point and the polar coordinate system is θ. Substitute the quadratic coefficient a", the constant term c" of the parabola, and the included angle θ between the pixel point and the polar coordinate system into Formula 5 to calculate the corresponding first distance threshold R.

[0141]

[0142] For example, in combination with Figure 4c As shown in the schematic diagram, when the coordinate position (x1", y1") of the pixel point m in the rectangular coordinate system x1"O1"y1" and the included angle θ1 between the pixel point m and the polar coordinate system are known, the first distance r1 between the pixel point m and the pole can be calculated according to Formula 4. And when the quadratic coefficient a2", the constant term c2" of the parabola 2" corresponding to the pixel point m, and the included angle θ1 between the pixel point m and the polar coordinate system are known, the corresponding first distance threshold R1 can be calculated according to Formula 5.

[0143] Adopt the above calculation method to obtain the first distance r of each pixel point in the simulated eye region, and screen out multiple pixel points whose first distance r does not exceed the set first distance threshold R; determine the region of the multiple pixel points in the simulated eye region as the inner contour region.

[0144] By performing step 203, align each second eye key point included in the simulated eye region with each first eye key point included in the actual eye region one by one, and align the contour shape of the simulated eye region with the contour shape of the actual eye region one by one, so that the eye offset direction, contour shape, and eye size of the simulated eye region are more consistent with the actual eye region.

[0145] Then, according to the eye state of the actual eye region, corresponding rendering materials are extracted from the simulated eye region. Finally, in the image to be processed, each second eye key point included in the rendering materials is respectively subjected to fusion rendering processing with the corresponding first eye key point to obtain the target image as shown in Figure 4d ; Among them, each time a fusion rendering process is performed for a second eye key point, it is determined based on the color of the rendering material of the second eye key point, the pixel point information of a corresponding first eye key point, and the skin color fusion coefficient. The skin color fusion coefficient is determined based on the skin color of the eye region of the actual eye region and the color of the corresponding rendering material.

[0146] As shown in Formula 6, when fusing and rendering the rendering material with a matte texture, the embodiments of the present application also take into account the influence of ambient light on the color and the color adaptability between the human face skin color and the rendering material, preventing the rendering material from being too obtrusive on the human face and improving the overall rendering effect.

[0147] Among them, the pixel point information of a first eye key point includes Src.rgb and Src_y. Src.rgb represents the three-primary color pixel values of a first eye key point, and Src_y represents the brightness value of the first eye key point, which is calculated according to the formula Src y =Src.rgb*[0.299,0.587,0.114] T obtained. And LightAdj(Src_y) represents appropriately adjusting the brightness value of the first eye key point according to the lighting condition of the current first eye key point, so that the fusion degree of the rendering material can be reduced in the case of overexposure. The specific adjustment method can be a piecewise function, polynomial adjustment or other adjustment methods, which are not limited herein. T.rgb represents the rendering material pixel value of the corresponding second eye key point, γ represents the fusion degree adjusted by the user, and W represents the skin color fusion coefficient. W is calculated according to the formula W = F(Src_y), and the purpose is to retain more texture details in the figure by appropriately adjusting the brightness value of the first eye key point.

[0148] Output = Src.rgb*(1 - γ)*LightAdj(Src_y)+(T.rgb*W + Src.rgb*T.rgb*(1 - W))*γ*LightAdj(Src_y) Formula 6;

[0149] In the embodiments of the present application, a variety of rendering materials with different textures are provided for users. Users can select any rendering material as virtual makeup material according to their own needs and preferences to obtain a target image containing the corresponding simulated eye makeup. For example, the above-mentioned rendering materials with single-color matte texture and multi-color matte texture, as well as the rendering materials with fine sparkle texture to be introduced below. Briefly speaking, the physical terminal device can add rendering materials with fine sparkle texture such as pearlescent light and metallic light to the eye area of the target image based on the texture image containing the fine sparkle effect, and generate a target image as shown in Figure 4e so that when the user's eyes deform, randomly flashing points are generated to improve the aesthetic feeling of the whole image.

[0150] Specifically, at least one eye pixel point is randomly selected from the eye area of the target image, and the following operations are respectively performed on the at least one eye pixel point to obtain a target image containing the fine sparkle effect: a texture pixel point is randomly selected on a preset texture image containing the fine sparkle effect, and based on the pixel value of an eye pixel point and the pixel value of this texture pixel point, the pixel value of this eye pixel point is updated.

[0151] As shown in Formula 7, in the embodiments of the present application, the strong light fusion method is first adopted to determine the pixel value of the updated eye pixel point, and then relevant random numbers are generated based on the coordinate position of the eye pixel point, and based on the random numbers, a random flashing effect generated when the pixel points of the eye part are displaced is simulated. Among them, Src.rgb represents the three-primary-color pixel value of a first eye key point, Shimmer.rgb represents the three-primary-color pixel value of a texture pixel point, Hardlight refers to the strong light fusion method, and Random(x', y') represents the randomly generated flashing point. In addition to the way of randomly adding flashing points, the way of periodically adding flashing points can also be adopted to add rendering materials with fine sparkle texture such as pearlescent light and metallic light to the eye area of the target image.

[0152] S C = Hardlight(Src.rgb, Shimmer.rgb)+Random(x', y') Formula 7;

[0153] After it is detected that the actual eye area of the user deforms, the contour shape of the standard eye area will be readjusted based on the new first eye key points after the deformation to obtain a new simulated eye area; then the corresponding rendering materials are extracted from the new simulated eye area, and through performing fusion rendering processing, a target image containing the virtual makeup effect is obtained.

[0154] Referring to Figure 5 the schematic flow chart shown, the image optimization rendering method provided by the embodiments of the present application is applied to provide real-time virtual makeup services for users in the live broadcast scenario.

[0155] S501: The physical terminal device responds to an image acquisition instruction triggered by a user, calls the camera to obtain the acquired video stream, and selects any frame of the image from the video stream as the image to be processed;

[0156] S502: The physical terminal device inputs the image to be processed into the eye region recognition model to obtain the actual eye region in the image to be processed;

[0157] S503: The physical terminal device obtains the standard eye region including the virtual makeup effect;

[0158] S504: The physical terminal device adjusts the shape of the standard eye region based on each first eye key point included in the actual eye region to obtain a simulated eye region including each second eye key point, where each first eye key point corresponds to each second eye key point one by one;

[0159] S505: The physical terminal device extracts the corresponding rendering materials from the simulated eye region based on the eye state corresponding to the actual eye region, and performs a fusion rendering process on each second eye key point included in the rendering materials and the corresponding first eye key point in the image to be processed to obtain a target image including the simulated matte eye makeup;

[0160] S506: The physical terminal device adds rendering materials with a fine flash texture to the eye region of the target image based on the texture image including the fine flash effect to generate a target image including the simulated fine flash eye makeup;

[0161] S507: The physical terminal device determines whether the actual eye region has deformed. If so, it returns to step 504; otherwise, it ends the entire process.

[0162] The image optimization rendering solution provided by the embodiments of the present application adopts a curve fitting modeling solution, which can better simulate the morphological changes of the eyes and make the constructed simulated eye region more realistic; and the curve fitting modeling solution, combined with the eye positioning solution based on the eye corner key points, can obtain a more stable rendering range, which is beneficial to improving the overall rendering effect.

[0163] The embodiments of the present application also consider the influence of ambient light on color, as well as the color adaptability between human skin color and rendering materials. When performing fusion rendering processing on images of users with different skin colors, images with more realistic and natural rendering effects can be obtained, preventing the situation that the rendering materials are too prominent on the human face. This not only improves the overall rendering effect but also enhances the robustness of the image optimization rendering scheme. In addition, the embodiments of the present application also provide users with a variety of rendering materials with different textures. Users can select any rendering material as the simulated eye makeup material according to their own needs and preferences, obtain a target image containing the corresponding simulated eye makeup, enhance the image beauty of users in scenarios such as live broadcasts and selfies, reduce the redundant operations when users edit portrait pictures, lower the operation difficulty when users beautify portrait images, and further improve the platform activity of users and the user usage volume of the platform.

[0164] Based on the same inventive concept as the above method embodiment, the embodiments of the present application also provide a structural schematic diagram of an apparatus for image optimization rendering. As Figure 6 shown, the apparatus 600 may include:

[0165] An acquisition unit 601, configured to obtain an image to be processed and determine the actual eye region in the image to be processed;

[0166] Obtain a preset standard eye region including a virtual makeup effect. The standard eye region is determined based on a standard face image, and the standard face image is obtained by averaging various types of face templates and is a face image in a frontal state;

[0167] A processing unit 602, configured to perform shape adjustment on the standard eye region based on each first eye key point included in the actual eye region to obtain a simulated eye region including each second eye key point, where each first eye key point and each second eye key point correspond one by one;

[0168] A rendering unit 603, configured to extract corresponding rendering materials from the simulated eye region based on the eye state corresponding to the actual eye region, and perform fusion rendering processing on each second eye key point included in the rendering materials with the corresponding first eye key point in the image to be processed to obtain a target image.

[0169] Optionally, the processing unit 602 is configured to:

[0170] Determine the deviation information between the actual eye region and the standard eye region based on each first eye key point included in the actual eye region, and adjust the offset angle of the standard eye region based on the deviation information;

[0171] Adjust the eye size of the rotated standard eye region according to the eye size of the actual eye region to obtain a scaled and adjusted standard eye region;

[0172] Update the contour of the standard eye region after rotation and scaling adjustment to obtain a simulated eye region.

[0173] Optionally, the processing unit 602 is configured to:

[0174] Determine the length ratio value between the second eye length of the rotated standard eye region and the first eye length of the actual eye region as the scaling ratio value of the rotated standard eye region;

[0175] Based on the scaling ratio value, adjust the eye size of the rotated standard eye region to obtain a scaled and adjusted standard eye region.

[0176] Optionally, the rendering unit 603 is configured to:

[0177] If the actual eye region corresponds to an open-eye state, determine the internal contour region in the simulated eye region based on the first distance of each pixel point in the simulated eye region and a set first distance threshold, and remove the internal contour region from the simulated eye region, and extract the region corresponding to the open-eye state from the simulated eye region as the rendering material;

[0178] If the actual eye region corresponds to a closed-eye state, extract the region corresponding to the closed-eye state from the simulated eye region as the rendering material.

[0179] Optionally, the rendering unit 603 determines the eye state corresponding to the actual eye region in the following manner:

[0180] Obtain the eyelid distance between the upper eyelid and the lower eyelid in the actual eye region;

[0181] If the eyelid distance exceeds a set second distance threshold, it is determined that the actual eye region corresponds to an open-eye state;

[0182] Otherwise, it is determined that the actual eye region corresponds to a closed-eye state.

[0183] Optionally, the rendering unit 603 is configured to:

[0184] Obtain the first distance of each pixel point in the simulated eye region, and filter and obtain a plurality of pixel points whose first distance does not exceed a set first distance threshold;

[0185] Determine the region of the plurality of pixel points in the simulated eye region as the internal contour region.

[0186] Optionally, the rendering unit 603 is configured to:

[0187] In the image to be processed, each second eye key point included in the rendering material is respectively subjected to fusion rendering processing with the corresponding first eye key point. Among them, each time a fusion rendering process is performed for one second eye key point, it is determined based on the rendering material color of one second eye key point, the pixel point information of the corresponding first eye key point, and the skin color fusion coefficient. The skin color fusion coefficient is determined based on the skin color of the actual eye area and the rendering material color corresponding thereto.

[0188] Optionally, after obtaining the target image, the rendering unit 603 is further configured to:

[0189] Randomly select at least one eye pixel point from the eye area of the target image;

[0190] Perform the following operations respectively on at least one eye pixel point to obtain a target image including a fine flash effect:

[0191] Randomly select a texture pixel point on a preset texture image including a fine flash effect;

[0192] Update the pixel value of one eye pixel point based on the pixel value of one eye pixel point and the pixel value of the one texture pixel point.

[0193] Optionally, the acquisition unit 601 obtains the image to be processed in any of the following ways:

[0194] In response to an image acquisition instruction triggered by a target object, obtain a frame of image collected from an image acquisition device, and use the frame of image as the image to be processed;

[0195] In response to an image acquisition instruction triggered by a target object, obtain a video stream collected from an image acquisition device, and select any frame of image from the video stream as the image to be processed.

[0196] For the convenience of description, the above parts are divided into various modules (or units) according to functions and described separately. Of course, when implementing the present application, the functions of the various modules (or units) can be implemented in the same or multiple software or hardware.

[0197] After introducing the method and device for image optimization rendering of the exemplary embodiment of the present application, next, a computer device according to another exemplary embodiment of the present application is introduced.

[0198] Those skilled in the art of the relevant technical field can understand that various aspects of the present application can be implemented as a system, a method, or a program product. Therefore, various aspects of the present application can be specifically implemented in the following forms, namely: a complete hardware implementation manner, a complete software implementation manner (including firmware, microcode, etc.), or an implementation manner combining hardware and software aspects, which can be collectively referred to as "circuit", "module", or "system" here.

[0199] Based on the same inventive concept as the above method embodiments, an embodiment of the present application also provides a computer device. Refer to Figure 7 As shown, the computer device 700 may at least include a processor 701 and a memory 702. Among them, the memory 702 stores program code. When the program code is executed by the processor 701, the processor 701 is caused to execute the steps of any of the above image optimization rendering methods.

[0200] In some possible implementation manners, the computing device according to the present application may at least include at least one processor and at least one memory. Among them, the memory stores program code. When the program code is executed by the processor, the processor is caused to execute the steps in the image optimization rendering method according to various exemplary embodiments of the present application described above in this specification. For example, the processor may execute the steps as shown in Figure 2a shown in.

[0201] Next, refer to Figure 8 to describe the computing device 800 according to this embodiment of the present application. Figure 8 The computing device 800 is only an example and should not impose any limitations on the functions and usage scope of the embodiments of the present application.

[0202] As Figure 8 shown, the computing device 800 is presented in the form of a general-purpose computing device. The components of the computing device 800 may include but are not limited to: the above at least one processing unit 801, the above at least one storage unit 802, and a bus 803 connecting different system components (including the storage unit 802 and the processing unit 801).

[0203] The bus 803 represents one or more of several types of bus structures, including a memory bus or a memory controller, a peripheral bus, a processor, or a local bus using any of the multiple bus structures.

[0204] The storage unit 802 may include a readable medium in the form of a volatile memory, such as a random access memory (RAM) 8021 and / or a cache storage unit 8022, and may further include a read-only memory (ROM) 8023.

[0205] The storage unit 802 may also include a program / utility 8025 having a set (at least one) of program modules 8024. Such program modules 8024 include but are not limited to: an operating system, one or more application programs, other program modules, and program data. The implementation of a network environment may be included in each or some combination of these examples.

[0206] The computing device 800 may also communicate with one or more external devices 804 (such as a keyboard, a pointing device, etc.), and may also communicate with one or more devices that enable a user to interact with the computing device 800, and / or communicate with any device that enables the computing device 800 to communicate with one or more other computing devices (such as a router, a modem, etc.). Such communication may be carried out through the input / output (I / O) interface 805. Moreover, the computing device 800 may also communicate with one or more networks (such as a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through the network adapter 806. As shown in the figure, the network adapter 806 communicates with other modules for the computing device 800 through the bus 803. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the computing device 800, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0207] Based on the same inventive concept as the above method embodiments, various aspects of the image optimization rendering method provided in this application can also be implemented in the form of a program product, which includes program code. When the program product runs on a computer device, the program code is used to cause the computer device to execute the steps in the image optimization rendering method according to various exemplary embodiments of this application described above in this specification. For example, an electronic device can execute the steps as shown in Figure 2a the figure.

[0208] The program product may adopt any combination of one or more readable media. The readable media may be a readable signal medium or a readable storage medium. The readable storage medium may, for example, be but is not limited to an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples (non-exhaustive list) of the readable storage medium include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0209] Although the preferred embodiments of this application have been described, those skilled in the art can make additional changes and modifications to these embodiments once they learn the basic creative concept. Therefore, the appended claims are intended to be construed as including the preferred embodiments as well as all changes and modifications that fall within the scope of this application.

[0210] Obviously, those skilled in the art can make various changes and modifications to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalent technologies, this application is also intended to cover these modifications and variations.

Claims

1. A method for optimizing and rendering an image, characterized in that, Including: Obtain an image to be processed and determine the actual eye region in the image to be processed; Obtain a preset standard eye region including virtual makeup effects, where the standard eye region is determined based on a standard face image, and the standard face image is obtained by averaging various types of face templates and is a face image in a frontal view state; Based on each first eye key point included in the actual eye region, determine the deviation information between the actual eye region and the standard eye region, and adjust the offset angle of the standard eye region based on the deviation information; Adjust the eye size of the rotated standard eye region according to the eye size of the actual eye region to obtain a scaled and adjusted standard eye region; Update the contour shape of the scaled and adjusted standard eye region to obtain a simulated eye region including each second eye key point, where each of the first eye key points and each of the second eye key points corresponds one by one; Based on the eye state corresponding to the actual eye region, extract corresponding rendering materials from the simulated eye region, and in the image to be processed, perform a fusion rendering process on each second eye key point included in the rendering materials with the corresponding first eye key point to obtain a target image.

2. The method according to claim 1, characterized in that The step of adjusting the eye size of the rotated standard eye region according to the eye size of the actual eye region to obtain a scaled and adjusted standard eye region includes: Determine the length ratio value between the second eye length of the rotated standard eye region and the first eye length of the actual eye region as the scaling ratio value of the rotated standard eye region; Based on the scaling ratio value, adjust the eye size of the rotated standard eye region to obtain the scaled and adjusted standard eye region.

3. The method according to claim 1, wherein The step of extracting corresponding rendering materials from the simulated eye region based on the eye state corresponding to the actual eye region includes: If the actual eye region corresponds to an open-eye state, based on the first distance of each pixel point in the simulated eye region and a set first distance threshold, determine the internal contour region in the simulated eye region, remove the internal contour region from the simulated eye region, and extract the region corresponding to the open-eye state from the simulated eye region as the rendering material; If the actual eye region corresponds to a closed-eye state, extract the region corresponding to the closed-eye state from the simulated eye region as the rendering material.

4. The method according to claim 3, characterized in that Determine the eye state corresponding to the actual eye region by the following method: Obtain the eyelid distance between the upper eyelid and the lower eyelid in the actual eye region; If the eyelid distance exceeds a set second distance threshold, determine that the actual eye region corresponds to an open-eye state; Otherwise, determine that the actual eye region corresponds to a closed-eye state.

5. The method according to claim 3, characterized in that The step of determining the internal contour region in the simulated eye region based on the first distance of each pixel point in the simulated eye region and a set first distance threshold includes: Obtain the first distance of each pixel point in the simulated eye region, and screen to obtain a plurality of pixel points whose first distance does not exceed a set first distance threshold; Determine the region of the plurality of pixel points in the simulated eye region as the inner contour region.

6. The method according to claim 1, wherein In the to-be-processed image, performing fusion rendering processing on each second eye key point included in the rendering material with a corresponding first eye key point to obtain a target image, including: In the to-be-processed image, performing fusion rendering processing on each second eye key point included in the rendering material with a corresponding first eye key point. Wherein, each time a fusion rendering process is performed for one second eye key point, it is determined based on the rendering material color of the one second eye key point, the pixel point information of the corresponding one first eye key point, and the skin color fusion coefficient, and the skin color fusion coefficient is determined based on the eye region skin color of the actual eye region and the corresponding rendering material color.

7. The method according to claim 1, characterized in that, After obtaining the target image, the method further includes: Randomly select at least one eye pixel point from the eye region of the target image; Perform the following operations on the at least one eye pixel point respectively to obtain a target image including a fine flash effect: Randomly select a texture pixel point on a preset texture image including a fine flash effect; Update the pixel value of the one eye pixel point based on the pixel value of the one eye pixel point and the pixel value of the one texture pixel point.

8. The method according to any one of claims 1 to 7, characterized in that, The to-be-processed image is obtained by any of the following methods: In response to an image acquisition instruction triggered by a target object, obtain a frame of image acquired from an image acquisition device, and use the frame of image as the to-be-processed image; In response to the image acquisition instruction triggered by the target object, obtain a video stream acquired from an image acquisition device, and select any frame of image from the video stream as the to-be-processed image.

9. An apparatus for optimizing image rendering, characterized in that, Including: An acquisition unit, configured to obtain a to-be-processed image and determine an actual eye region in the to-be-processed image; Obtain a preset standard eye region including a virtual makeup effect, where the standard eye region is determined based on a standard face image, and the standard face image is obtained by averaging various types of face templates, and is a face image in a frontal state; A processing unit, configured to determine deviation information between the actual eye region and the standard eye region based on each first eye key point included in the actual eye region, and adjust the offset angle of the standard eye region based on the deviation information; Adjust the eye size of the rotated standard eye region according to the eye size of the actual eye region to obtain a scaled and adjusted standard eye region; Update the contour shape of the scaled and adjusted standard eye region to obtain a simulated eye region including each second eye key point, where each of the first eye key points and each of the second eye key points corresponds one by one; A rendering unit, configured to extract corresponding rendering materials from the simulated eye region based on the eye state corresponding to the actual eye region, and in the image to be processed, perform a fusion rendering process on each second eye key point included in the rendering materials with the corresponding first eye key point, to obtain a target image.

10. The device according to claim 9, characterized in that, The processing unit is configured to: Determine a length ratio value between the second eye length of the rotated standard eye region and the first eye length of the actual eye region as the scaling ratio value of the rotated standard eye region; Based on the scaling ratio value, adjust the eye size of the rotated standard eye region to obtain the scaled and adjusted standard eye region.

11. The device according to claim 9, characterized in that, The rendering unit is configured to: If the eye state corresponding to the actual eye region is the open-eye state, determine an internal contour region in the simulated eye region based on the first distance of each pixel point in the simulated eye region and a set first distance threshold, and remove the internal contour region from the simulated eye region, and extract a region corresponding to the open-eye state from the simulated eye region as the rendering material; If the eye state corresponding to the actual eye region is the closed-eye state, extract a region corresponding to the closed-eye state from the simulated eye region as the rendering material.

12. A computer device, characterized in that, It includes a processor and a memory. Among them, the memory stores program codes. When the program codes are executed by the processor, the processor is caused to execute the steps of the method according to any one of claims 1 to 8.

13. A computer-readable storage medium, characterized in that, It includes program codes. When the program product runs on a computer device, the program codes are used to cause the computer device to execute the steps of the method according to any one of claims 1 to 8.

Images