Image rendering processing method, device and equipment, and computer storage medium

Abstract

The present application discloses an image rendering processing method, apparatus and equipment and computer storage medium, relating to the field of artificial intelligence technology. In this method, when performing virtual eyebrow makeup on an image to be processed, the deformed area to be processed is determined based on the key points of the eyebrow area, and the deformed area to be processed is scaled and mapped to reduce the eyebrow area. When an eyebrow with a makeup effect is subsequently added, the eyebrow area can be well covered, thereby presenting a better makeup effect. In addition, when rendering the material, by establishing a pixel mapping relationship, each pixel can be rendered independently and in parallel, thereby improving the efficiency of image rendering and correspondingly improving the real-time performance of video makeup.

Description

Technical Field

[0001] The present application relates to the field of computer technology, in particular to the field of artificial intelligence technology, and provides an image rendering processing method, apparatus and device, and a computer storage medium. Background Art

[0002] Currently, in video processing scenarios, such as short video scenarios or live video broadcast scenarios, there is often a demand for automatic facial makeup. Eyebrows are an indispensable part of makeup and directly affect the quality of makeup effects.

[0003] In related technologies, there are solutions that can automatically apply eyebrow makeup through augmented reality. However, these solutions often directly overlay eyebrow templates on the eyebrow area of ​​the face, which may cause the eyebrow template to not match the original eyebrows. As a result, the eyebrow template cannot completely cover the original eyebrows, resulting in poor makeup effect. Based on this, users can only choose the appropriate eyebrow template based on their own eyebrow condition to cover the original eyebrows. When users want to try different eyebrow shapes, related technical solutions still have certain limitations and cannot achieve the best makeup effect. Summary of the Invention

[0004] The embodiments of the present application provide an image rendering processing method, apparatus, device, and computer storage medium for improving the makeup effect after image rendering and image rendering efficiency.

[0005] In one aspect, a method for image rendering is provided, the method comprising:

[0006] Performing facial key point recognition on a first image to be processed containing a human face, and identifying a plurality of first key points corresponding to a first actual eyebrow region of the human face from the first image to be processed;

[0007] Based on the multiple first key points, determining, in the first image to be processed, a deformed region to be processed that includes the first actual eyebrow region, and performing scaling mapping processing on each pixel point in the deformed region to be processed to obtain a second image to be processed that includes a scaled second actual eyebrow region;

[0008] Obtaining a target eyebrow template including a virtual makeup effect, and establishing a one-to-one pixel mapping relationship between the first actual eyebrow region and the simulated eyebrow region based on the plurality of first key points and a plurality of second key points corresponding to the simulated eyebrow region in the eyebrow template;

[0009] Based on the pixel mapping relationship, rendering materials of each pixel point are extracted from the simulated eyebrow area, and fusion rendering processing is performed on the corresponding pixel points in the second image to be processed to obtain a target image.

[0010] In one aspect, an image rendering processing device is provided, the device comprising:

[0011] a face recognition unit, configured to perform facial key point recognition on a first image to be processed containing a face, and identify a plurality of first key points corresponding to a first actual eyebrow region of the face from the first image to be processed;

[0012] a scaling mapping unit configured to determine, in the first image to be processed, a deformed region to be processed that includes the first actual eyebrow region based on the plurality of first key points, and perform scaling mapping processing on each pixel in the deformed region to be processed to obtain a second image to be processed that includes a scaled second actual eyebrow region;

[0013] a grid construction unit, configured to obtain a target eyebrow template including a virtual makeup effect, and establish a one-to-one pixel mapping relationship between the first actual eyebrow region and the simulated eyebrow region based on the plurality of first key points and a plurality of second key points corresponding to the simulated eyebrow region in the eyebrow template;

[0014] The texture fitting unit is used to extract rendering materials of each pixel point from the simulated eyebrow area based on the pixel mapping relationship, and perform fusion rendering processing on the corresponding pixel points in the second image to be processed to obtain a target image.

[0015] Optionally, the scaling mapping unit is specifically configured to:

[0016] Based on the coordinate values ​​of the plurality of first key points in the constructed reference coordinate system, determining, in the first image to be processed, reference points of the deformation area to be processed;

[0017] Based on the set shape model, the deformation area to be processed is determined in the first image to be processed with the reference point as the center.

[0018] Optionally, the scaling mapping unit is specifically configured to:

[0019] The pixel point corresponding to the average of the coordinate values ​​corresponding to the plurality of first key points is determined as the reference point.

[0020] Optionally, the scaling mapping unit is specifically configured to:

[0021] Determining the distances between the reference point and each first key point, determining a first distance having the largest value from the obtained distances, and determining a semi-major axis based on the first distances;

[0022] Selecting a second distance with a larger value from the distances between the reference point and the two nearest first key points, and determining the semi-minor axis based on the second distance;

[0023] An elliptical area centered at the reference point and enclosed by the semi-major axis and the semi-minor axis is determined as the deformation area to be processed.

[0024] Optionally, the scaling mapping unit is specifically configured to:

[0025] Determine an angle between a line connecting the reference point and a first key point corresponding to the first distance and a set coordinate axis in the reference coordinate system;

[0026] For each pixel in the image to be processed, perform the following operations:

[0027] For a pixel point, after rotating the pixel point by the angle value with the reference point as the center, determine the distance between the position of the rotated pixel point and the reference point;

[0028] If the distance between the position of the rotated pixel point and the reference point is less than a set distance threshold, the pixel point is determined to be a pixel point located in the deformation area to be processed.

[0029] Optionally, the scaling mapping unit is specifically configured to:

[0030] Determining the distances between the reference point and each first key point, and determining the distance with the largest value from the obtained distances;

[0031] A circular area with the reference point as the center and the maximum distance as the radius is determined as the deformation area to be processed.

[0032] Optionally, the scaling mapping unit is specifically configured to:

[0033] For each pixel in the deformed area to be processed, perform the following operations to obtain the second image to be processed:

[0034] For a pixel point, determining a scaled coordinate value of the pixel point based on the coordinate value of the pixel point and a set scaling mapping relationship; wherein, in the set scaling mapping relationship, the scaling ratio is negatively correlated with the distance between the reference points of the deformation area to be processed;

[0035] The pixel point at the location of the scaled coordinate value is updated using the pixel feature value of the one pixel point.

[0036] Optionally, the grid construction unit is specifically configured to:

[0037] Based on the multiple first key points, meshing the first actual eyebrow region to obtain a first mesh structure; wherein the multiple first key points are vertices of meshes in the first mesh structure, and one mesh corresponds to a sub-region of the first actual eyebrow region; and

[0038] Based on the plurality of second key points, meshing the simulated eyebrow region is performed to obtain a second mesh structure; wherein the plurality of second key points are vertices of meshes in the second mesh structure, and one mesh corresponds to a sub-region of the simulated eyebrow region, and the meshes in the first mesh structure correspond one-to-one to the meshes in the second mesh structure;

[0039] The pixel mapping relationship is established based on the grid correspondence relationship between the first grid structure and the second grid structure.

[0040] Optionally, the grid construction unit is specifically configured to:

[0041] Taking the multiple first key points and the at least one set auxiliary key point as vertices, and based on a set gridding processing method, performing gridding processing on the first actual eyebrow area to obtain the first grid structure;

[0042] The gridding processing method is determined based on a standard eyebrow region in a standard face image, and the standard face image is a face image in a frontal view state obtained by averaging various types of face templates.

[0043] Optionally, the apparatus further includes a video stream processing unit, configured to:

[0044] A video stream to be processed is obtained, and an image containing a human face is selected from the video stream to be processed as the image to be processed; and a target video stream containing a virtual makeup effect is obtained based on target images corresponding to each image to be processed.

[0045] In one aspect, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any one of the above methods when executing the computer program.

[0046] In one aspect, a computer storage medium is provided, on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the steps of any of the above methods are implemented.

[0047] In one aspect, a computer program product or computer program is provided, the computer program product or computer program comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps of any of the above methods.

[0048] In an embodiment of the present application, when performing virtual eyebrow makeup on the processed image, the deformed area to be processed is determined based on the key points of the eyebrow area, and the deformed area to be processed is scaled and mapped to reduce the eyebrow area. When the eyebrows with makeup effects are subsequently added, the eyebrow area can be well covered, thereby presenting a better makeup effect. In addition, when rendering the material, by establishing a pixel mapping relationship, each pixel can be rendered independently and in parallel, thereby improving the efficiency of image rendering and correspondingly improving the real-time performance of video makeup. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0050] Figure 1 Schematic diagram of application scenarios provided by embodiments of the present application;

[0051] Figure 2 A flowchart of an image rendering method according to an embodiment of the present invention;

[0052] Figure 3 A schematic diagram of key points of the eyebrow area provided in an embodiment of the present application;

[0053] Figure 4 A schematic diagram comparing images before and after scaling and mapping processing provided in an embodiment of the present application;

[0054] Figure 5 A schematic diagram of the deformation region to be processed of the elliptical model provided in an embodiment of the present application;

[0055] Figure 6 A schematic diagram of a process for determining whether a pixel point A is an inner point of an ellipse according to an embodiment of the present application;

[0056] Figure 7 A schematic diagram of the deformation area to be processed of the circular model provided in an embodiment of the present application;

[0057] Figure 8A schematic diagram of pixel scaling and mapping processing provided in an embodiment of the present application;

[0058] Figure 9 A schematic diagram of the corresponding relationship of the constructed grid structure provided in an embodiment of the present application;

[0059] Figure 10 A grid structure diagram after adding auxiliary key points provided in an embodiment of the present application;

[0060] Figure 11 A schematic diagram of the process of image rendering in a video stream provided in an embodiment of the present application;

[0061] Figure 12 A schematic diagram of the structure of an image rendering processing device provided in an embodiment of the present application;

[0062] Figure 13 A schematic diagram of the structure of a computer device provided in an embodiment of the present application;

[0063] Figure 14 A schematic diagram of the structure of a computing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0064] In order to make the purpose, technical solutions and advantages of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application. In the absence of conflict, the embodiments in the present application and the features in the embodiments can be combined with each other in any way. In addition, although a logical order is shown in the flow chart, in some cases, the steps shown or described can be performed in an order different from that here.

[0065] To facilitate understanding of the technical solutions provided in the embodiments of the present application, some key terms used in the embodiments of the present application are explained here:

[0066] Facial key points: refers to the pixels in a facial image that can describe facial features. Generally speaking, they can include pixels of facial contour features and facial features. In the embodiment of this application, key points mainly refer to key points for eyebrows. Based on the key points of eyebrows, the contour features of the eyebrows can be described. Generally speaking, the detection of facial key points can be based on a facial key point detection model. The more common facial key point models can be a facial key point detection model based on 106 key points, a facial key point detection model based on 256 key points, and a facial key point detection model based on 68 key points.

[0067] Scaling mapping: also known as liquefaction deformation, refers to the use of liquefaction deformation processing to perform a certain deformation processing on the eyebrow area and surrounding areas, making the eyebrow area appear smaller on the face.

[0068] Standard face: It is an averaged face obtained based on statistics of a large amount of facial data, which can represent the characteristics of most faces.

[0069] The technical solutions of the embodiments of this application involve artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to achieve the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is the study of the design principles and implementation methods of various intelligent machines, so that machines have the functions of perception, reasoning and decision-making.

[0070] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, interactive systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, machine learning / deep learning, autonomous driving, and smart transportation. With the development and advancement of AI, research and application are being expanded across a wide range of fields, including smart homes, smart customer service, virtual assistants, smart speakers, smart marketing, smart wearables, driverless cars, autonomous driving, drones, robotics, smart healthcare, connected vehicles, autonomous driving, and smart transportation. With further technological advancements, AI will be applied in even more areas, playing an increasingly important role.

[0071] Machine learning (ML) is at the core of artificial intelligence (AI) and the fundamental way to make computers intelligent. Its applications span all areas of AI. At the core of ML is deep learning, a technology that enables machine learning. Machine learning generally includes techniques such as deep learning, reinforcement learning, transfer learning, inductive learning, artificial neural networks, and self-learning. Deep learning includes technologies such as convolutional neural networks (CNNs), deep belief networks, recurrent neural networks, autoencoders, and generative adversarial networks.

[0072] Computer vision (CV) is a multidisciplinary field that integrates computer science, signal processing, physics, applied mathematics, statistics, neurophysiology, and other disciplines. It is also a challenging and important research area in the scientific field. Computer vision is the study of how to make machines "see." More specifically, it refers to the use of various imaging systems such as cameras and computers to replace human visual organs to perform machine vision processing such as target identification, tracking, and measurement. Through further graphics processing, the collected images are processed into images more suitable for human observation or transmission to instrument detection.

[0073] Computer vision, as a scientific discipline, aims to enable computers to observe and understand the world through vision, just as humans do, by studying relevant theories and technologies. This allows the development of artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, optical character recognition (OCR), video processing, video semantic understanding, video content / action recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping, autonomous driving, and smart transportation. In addition, computer vision technologies include common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0074] Augmented Reality (AR) technology simulates physical information (such as visual information, sound information, taste information, tactile information, etc.) that is difficult to experience within a certain time and space range in the real world through computers and other scientific and technological means, and then superimposes virtual information on the real world. It realizes that in the same picture or space, two different information, the real world environment and virtual world objects, are displayed to the user at the same time, achieving a sensory experience beyond reality.

[0075] The embodiments of this application mainly involve deep learning and computer vision technologies in artificial intelligence technology, that is, a face detection model is obtained by deep learning, and the trained face detection model is used to realize the detection of eyebrow key points in the face, so as to assist in the subsequent combination of computer vision technology for eyebrow-related processing. The specific process will be explained through subsequent embodiments and will not be elaborated on here.

[0076] The following is a brief introduction to the design concept of the embodiments of this application.

[0077] Currently, in video processing scenarios, such as short videos or live video streaming, there is often a demand for automatic facial makeup. Eyebrows are an indispensable part of makeup. Beautiful eyebrows can effectively modify the face shape, enhance the three-dimensional effect of the face, and make facial expressions more vivid and lively, which directly affects the quality of the makeup. Generally speaking, in real life, users often achieve this by manually applying makeup to their eyebrows. First, they need to determine the eyebrow shape based on the eyebrow conditions, and then use eyebrow drawing tools to apply makeup. This requires users to have a certain level of makeup experience, is time-consuming and consumes materials, and it is not easy to change different eyebrow shapes in a short period of time.

[0078] In the related art, there is a solution that can automatically apply makeup to eyebrows through augmented reality. That is, the eyebrows are first identified through detection or segmentation methods to obtain the eyebrow area, and then the pure eyebrow template is affine transformed to obtain a deformed template, which is then overlaid on the original eyebrows to obtain the eyebrows after makeup. Alternatively, the original eyebrows are subtracted and filled with the color of the skin around the eyebrows, and then the texture of the eyebrow template is pasted.

[0079] However, these solutions often directly superimpose eyebrow templates on the eyebrow area of ​​the face, which may cause the eyebrow template to not match the original eyebrows, making it impossible for the eyebrow template to completely cover the original eyebrows, resulting in poor makeup effect. Based on this, users can only choose a suitable eyebrow template according to their own eyebrow conditions to cover the original eyebrows. When users want to try different eyebrow shapes, the relevant technical solutions still have certain limitations and cannot show a better makeup effect. The method of removing the original eyebrows still has challenges when filling the surrounding skin. If the filling is not done properly, it may cause abnormal display effects around the eyebrows, such as uneven skin color, and thus cannot show a better makeup effect.

[0080] Based on this, an embodiment of the present application provides an image rendering processing method, in which, when performing virtual eyebrow makeup on the image to be processed, the deformed area to be processed is determined based on the key points of the eyebrow area, and the deformed area to be processed is scaled and mapped to reduce the eyebrow area. When the eyebrows with makeup effects are subsequently added, the eyebrow area can be well covered, thereby presenting a better makeup effect. In addition, when rendering the material, by establishing a pixel mapping relationship, each pixel can be rendered independently and in parallel, thereby improving the efficiency of image rendering and correspondingly improving the real-time performance of video makeup.

[0081] After introducing the design concepts of the embodiments of the present application, the following briefly introduces the application scenarios to which the technical solutions of the embodiments of the present application can be applied. It should be noted that the application scenarios introduced below are only used to illustrate the embodiments of the present application and are not limiting. In the specific implementation process, the technical solutions provided by the embodiments of the present application can be flexibly applied according to actual needs.

[0082] The solution provided by the embodiment of the present application can be applied to most scenarios where image rendering and processing are required, such as live broadcast platforms, content sharing platforms, and other scenarios where image beautification is required. It can also be applied to daily scenarios such as video calls, simulated makeup trials at offline counters or online e-commerce platforms. Figure 1 As shown, it is an application scenario diagram provided by an embodiment of the present application, in which a terminal device 101 and a background server 102 may be included.

[0083] The terminal device 101 may be, for example, a mobile phone, a tablet computer (PAD), a laptop computer, a desktop computer, a smart TV, or a smart wearable device. The terminal device 101 may be installed with an application capable of image beautification, such as a social application, a beauty application, or a photography application.

[0084] The backend server 102 may be a backend server corresponding to an application installed on the terminal device 101. The backend server 102 may be, for example, an independent physical server, or a server cluster or distributed system composed of multiple physical servers. The backend server 102 may also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms, but is not limited thereto.

[0085] The backend server 102 may include one or more processors 1021, a memory 1022, and an I / O interface 1023 for interacting with a terminal. In addition, the backend server 102 may also be configured with a database 1024, which may be used to store model data and image rendering processing parameter data. The memory 1022 of the backend server 102 may also store program instructions for the image rendering processing method provided in the embodiment of the present application. When these program instructions are executed by the processor 1021, they may be used to implement the steps of the image rendering processing method provided in the embodiment of the present application, so as to perform image rendering processing on the image to be processed and obtain a target image with an eyebrow makeup effect.

[0086] In one possible implementation, the user provides an image to be processed in an application installed on the terminal device 101, such as selecting an image from an album or reshooting an image, and then the terminal device 101 can upload the image to be processed to the backend server 102. The backend server 102 processes the image based on the image rendering processing method provided in the embodiment of the present application to obtain a target image with a virtual makeup effect, and then returns it to the terminal device 101 for display.

[0087] In another possible implementation, the user uses an application installed on the terminal device 101 to capture video and transmits the video stream to the backend server 102. The backend server 102 processes the video frames containing the face in the video stream based on the eyebrow template selected by the user and returns them to the terminal device 102 in real time, so that the terminal device 102 can present the video with makeup effects in real time during the video capture process. In addition, in some scenarios, such as live video broadcasting or video call scenarios, the backend server 102 can also send the processed video frames to the terminal devices 102 of other users, so that the terminal devices 102 of other users can also present the video with status.

[0088] In specific implementation, when the processing capability of the terminal device 101 allows, the image rendering process can also be performed on the terminal device 101, and the embodiment of the present application does not impose any restrictions on this.

[0089] The terminal device 101 and the backend server 102 can be directly or indirectly connected to each other through one or more networks 103. The network 103 can be a wired network or a wireless network, for example, a mobile cellular network or a Wireless Fidelity (WIFI) network, or other possible networks, which are not limited in the present embodiment.

[0090] It should be noted that, in the embodiment of the present application, the number of terminal devices 101 can be one or more, that is, there is no limit to the number of terminal devices 101.

[0091] Of course, the method provided in the embodiment of the present application is not limited to Figure 1 The application scenarios shown can also be used in other possible application scenarios, and the embodiments of this application are not limited thereto. Figure 1 The functions that can be implemented by each device in the application scenario shown will be described in subsequent method embodiments and will not be described in detail here.

[0092] See Figure 2 , is a flow chart of the image rendering processing method provided by the embodiment of the present application, which can be performed by Figure 1The backend server 102 or the terminal device 101 is used to execute the method, and the process of the method is described as follows.

[0093] Step 201: performing facial key point recognition on a first image to be processed containing a human face, and identifying a plurality of first key points corresponding to a first actual eyebrow region of the human face from the first image to be processed.

[0094] In an embodiment of the present application, the first image to be processed can be a single image or a video frame in a video. When the first image to be processed is a single image, the image can be a captured image selected by the user from the photo album, or an image re-shot by the user. When the first image to be processed is a video frame in a video, the video can be a captured video selected by the user from the photo album, or a video recorded by the user in real time.

[0095] In actual applications, when a user requests to add a virtual makeup effect to a video, face recognition can be performed on the video to determine whether there are any images to be processed containing faces, that is, images to be processed that need to be rendered, and then subsequent rendering processing can be performed on the images to be processed that contain faces in the video.

[0096] In order to perform targeted processing on the eyebrow area later, it is first necessary to identify the actual area where the eyebrow is located from the image to be processed. For the sake of distinction, the original eyebrow area in the image to be processed is called the first actual eyebrow area, and the key point in the image to be processed is called the first key point. Then, to identify the area where the eyebrow is located, it is necessary to identify the required multiple first key points from the image to be processed. The area surrounded by these multiple first key points is the first actual eyebrow area.

[0097] Specifically, each first key point can be identified by a pre-trained facial key point model, and any possible facial key point model can be used, such as a facial key point model based on 106 points, a facial key point model based on 68 points, and a facial key point model based on 256 points. Here, a facial key point model based on 106 points is taken as an example, see Figure 2 As shown, there are multiple first key points of the first actual eyebrow area obtained based on the 106-point facial key point model, namely Figure 3 Among the key points 1 to 19 shown, key points 1 to 5 and 15 to 18 are the key points of the left eyebrow, and key points 6 to 14 are the key points of the right eyebrow.

[0098] In specific implementation, the facial key point model needs to be trained in advance before it can be put into practical use. During training, a large number of training samples that have been pre-labeled with facial key points can be used to train the constructed facial key point model in multiple cycles until the number of iterations reaches the set number of conditions or the accuracy of the facial key point model meets the set conditions. Among them, the training samples can be obtained from the open source training data set, or the collected facial images can be labeled to form training samples. The facial key point model can use any possible deep neural network, and the embodiments of the present application do not limit this.

[0099] Step 202: Based on multiple first key points, in the first image to be processed, a deformation area to be processed including the first actual eyebrow area is determined, and scaling mapping processing is performed on each pixel point in the deformation area to be processed to obtain a second image to be processed including the scaled second actual eyebrow area.

[0100] In an embodiment of the present application, in order to make the eyebrows in the target image after rendering more realistic and fitting, the original eyebrows, that is, the first actual eyebrow area mentioned above, are liquefied and deformed, so that the first actual eyebrow area is scaled to obtain a second image to be processed containing the scaled second actual eyebrow area.

[0101] See also Figure 4 The figure shows a comparison diagram of images before and after scaling and mapping processing. Before scaling and mapping processing, the first actual eyebrow area in the first generation processed image is consistent with the original eyebrow size of the human face. After scaling and mapping processing, the first actual eyebrow area is reduced in size in the human face, thereby obtaining the second image to be processed, see Figure 4 As shown, in the second image to be processed, the rest of the parts except the eyebrow area have not changed, while the second actual eyebrow area is obviously smaller than the first actual eyebrow area, and the eyebrows in the second actual eyebrow area are obviously smaller. In this way, after the eyebrows in the eyebrow template are added later, it is easy to cover the shrunken eyebrows, so that the eyebrow makeup effect after adding the eyebrow template can be correspondingly improved.

[0102] Generally speaking, image processing is performed on a region. It is difficult to process only the eyebrow region without involving the surrounding areas. In order to make the effect after processing more natural, the surrounding areas also need to be processed. Before performing scaling mapping processing, it is necessary to determine the area to be scaled and mapped, that is, the deformed area to be processed.

[0103] In an embodiment of the present application, when determining the deformed area to be processed, the reference point of the deformed area to be processed can be determined in the first image to be processed based on the coordinate value of the first key point corresponding to the above-mentioned first actual eyebrow area in the constructed reference coordinate system, and then, based on the set shape model, the deformed area to be processed can be determined in the first image to be processed with the reference point as the center.

[0104] The reference coordinate system is generally constructed based on the first image to be processed.

[0105] In a possible implementation, a reference coordinate system may be constructed with the center point of the first image to be processed as the original coordinate and the horizontal and vertical directions as the horizontal axis and the vertical axis respectively.

[0106] In another possible implementation, a reference coordinate system may be constructed with one corner point of the first image to be processed as the original coordinate and the horizontal and vertical directions as the horizontal axis and the vertical axis respectively.

[0107] It should be noted that the above-mentioned deformation region to be processed is for a single eyebrow, that is, for a face, two deformation regions to be processed will be determined, one deformation region to be processed corresponds to one eyebrow, and then the eyebrows on both sides are processed separately.

[0108] In the embodiment of the present application, the pixel point corresponding to the average value of the coordinate values ​​of each first key point can be determined as the reference point of the deformation area to be processed, that is, the reference point of the deformation area to be processed can be expressed as follows:

[0109]

[0110] Among them, (x0, y0) is the coordinate value of the reference point, (x i ,y i ) is the coordinate value of the i-th first key point, and N is the total number of first key points corresponding to a single eyebrow.

[0111] Of course, other possible methods may also be used to calculate the reference points. For example, shape fitting may be performed based on each first key point, and the focus may be determined as the reference point. This embodiment of the present application does not limit this.

[0112] In the embodiment of the present application, the shape model may include, for example, an elliptical model, a circular model, or a triangular model, etc. Taking the elliptical model as an example, it means that the outer contour of the deformation area to be processed is an ellipse, and other shape models are similar.

[0113] Based on different shape models, the methods for determining the deformation area to be processed are different, which are introduced below.

[0114] (1) Elliptical model

[0115] When the shape model is an ellipse model, the distances between the reference point and each first key point may be calculated one by one, and a first distance with the largest value may be determined from the obtained distances, and the semi-major axis of the ellipse may be determined based on the first distances.

[0116] Taking the reference point as the average of the first key points as an example, generally speaking, the first key point at the end of the eyebrow should be the farthest from the reference point. Figure 5 As shown, the reference point is pixel point 0, and the point farthest from the reference point 0 is generally the first key point 1, so that the semi-major axis of the ellipse can be determined based on the distance between the reference point 0 and the first key point, and the straight line connecting the reference point 0 and the first key point is the straight line where the major axis of the ellipse is located.

[0117] In addition, the second distance with a larger value can be selected from the distances between the reference point and the two nearest first key points, and the semi-minor axis of the ellipse can be determined based on the second distance to obtain the deformation area to be processed. Figure 5 As shown in FIG, a schematic diagram of the constructed ellipse model is shown. The two first key points closest to the reference point 0 are 3 and 17. Then the distances between the reference point 0 and the first key points 3 and 17 can be calculated respectively, and the larger value can be selected as the reference distance of the semi-minor axis.

[0118] In one possible implementation, the first distance can be used as the semi-major axis of the ellipse, and the second distance can be determined as the semi-minor axis. Then, based on the reference point, the semi-major axis, and the semi-minor axis, the following ellipse equation can be determined:

[0119]

[0120] Wherein, d1 is the length of the semi-major axis of the ellipse, that is, the first distance mentioned above, and d2 is the length of the semi-minor axis of the ellipse, that is, the second distance mentioned above.

[0121] In another possible implementation, in order to avoid the situation where the ellipse cannot enclose the entire eyebrow area, a certain magnification factor can be set, that is, after magnifying the first distance and the second distance, the magnified first distance and the second distance are used as the semi-major axis and the semi-minor axis respectively to construct the elliptical area. Figure 5 This is specifically illustrated in the example. Furthermore, the ellipse equation can be expressed as follows:

[0122]

[0123] Among them, α and β are the magnifications corresponding to the semi-major axis and the semi-minor axis respectively. The values ​​of α and β can be the same or different. They can be set based on empirical values ​​or adjusted according to the actual eyebrow situation.

[0124] In the embodiment of the present application, after the ellipse model is constructed, the ellipse area can be used as the deformation area to be processed for scaling mapping, that is, Figure 5 The area enclosed by the ellipse is the deformation area to be processed and is subjected to scaling mapping processing.

[0125] Specifically, in order to know which pixels to scale and map later, it is necessary to use the elliptical area constructed above as a reference to further determine which pixels are within the elliptical area. Since the determination process for each pixel is similar, only pixel A is used as an example here. Figure 6 FIG. 1 is a flow chart showing a process of determining whether pixel point A is an inner point of an ellipse.

[0126] S1: Determine an angle between a line connecting the reference point and a first key point corresponding to a first distance and a set coordinate axis in a reference coordinate system.

[0127] Since the coordinates of each reference point, pixel point or first key point are all coordinate values ​​in the reference coordinate system, they need to be converted to the coordinate system of the ellipse for distance calculation.

[0128] S2: After rotating pixel point A by the angle value with the reference point as the center, determine the distance between the position of the rotated pixel point A and the reference point.

[0129] See also Figure 5 As shown, there is a certain angle θ between the line connecting the reference point and the first key point 1 and the horizontal axis. Therefore, when performing coordinate system conversion, the pixel point needs to be rotated. If the coordinate value of pixel point A in the reference coordinate system is (x a ,y a ), then the coordinate value after the rotation angle Θ is (x a ′,y a ′), the distance between pixel point A and reference point 0 can be defined as (here taking the first implementation above as an example):

[0130]

[0131] Where r is the distance between pixel point A and the center of the ellipse.

[0132] S3: Determine whether the distance between the position of the rotated pixel point A and the reference point is less than a set distance threshold.

[0133] S4: If the determination result of step S3 is no, determine that the pixel point A is a pixel point that does not belong to the deformation area to be processed.

[0134] S5: If the determination result of step S3 is yes, determine that pixel point A is a pixel point located in the deformation area to be processed.

[0135] Specifically, with respect to the above-mentioned distance definition, the distance threshold can be set to 1. That is, when r is less than 1, the pixel point A belongs to an inner point of the ellipse, that is, a pixel point in the deformation area to be processed that needs to be scaled and mapped. When r is greater than or equal to 1, the pixel point A belongs to an outer point of the ellipse or a point on the ellipse line, that is, a pixel point in the deformation area to be processed that does not need to be scaled and mapped.

[0136] (2) Circular model

[0137] When the shape model is a circular model, the distances between the reference point and each first key point can also be calculated one by one, and a first distance with the largest value can be determined from the obtained distances, and the circle radius can be determined based on the first distance.

[0138] Taking the reference point as the average of the first key points as an example, generally speaking, the first key point at the end of the eyebrow should be the farthest from the reference point. Figure 7 As shown, the reference point is pixel point 0, and the point farthest from the reference point 0 is generally the first key point 1, so the radius of the circle can be determined based on the distance between the reference point 0 and the first key point.

[0139] In a possible implementation, the first distance can be used as the radius of the circle. Figure 7 Specifically, the following circle equation can be determined based on the reference point and the circle radius:

[0140]

[0141] Wherein, d1 is the radius of the circle, that is, the first distance mentioned above.

[0142] In another possible implementation, a certain magnification factor can also be set, that is, after the first distance is magnified, the magnified first distance is used as the radius to construct a circular area. Furthermore, the ellipse equation can be expressed as follows:

[0143]

[0144] Likewise, the method of determining whether each pixel point is a point in a circular area is similar to the above-mentioned ellipse model, and thus will not be described in detail here.

[0145] It should be noted that the embodiments of the present application may also adopt any other possible shape models, and are not limited to this.

[0146] In the embodiment of the present application, scaling mapping processing is performed on each pixel point in the determined deformation area to be processed in the following manner, which is illustrated here by taking the first key point 1 in the deformation area to be processed as an example.

[0147] For the first key point 1, the scaled coordinate value of the first key point 1 can be determined based on the coordinate value of the first key point 1 and the set scaling mapping relationship, and then the pixel point at the location of the scaled coordinate value is updated with the pixel feature value of the first key point 1, so as to obtain the following: Figure 4 The second image to be processed shown includes a reduced second actual eyebrow area.

[0148] See also Figure 8 The figure below is a schematic diagram of the pixel scaling and mapping process. It can be seen that after the original eyebrow is scaled down, the first key point 1 on the original eyebrow is translated to the position of pixel 1'. In essence, the pixel value at the position of pixel 1' is replaced with the pixel feature value of the first key point 1.

[0149] Specifically, in the set scaling mapping relationship adopted in the embodiment of the present application, the scaling ratio is negatively correlated with the distance between the reference points of the deformed area to be processed. That is, the farther the pixel point is from the reference point of the deformed area to be processed, the more obvious the deformation is, and the closer the point is to the edge of the deformed area to be processed, the smaller the deformation is, until no deformation occurs at the edge of the deformed area to be processed. Therefore, any scaling mapping relationship that satisfies this deformation relationship is applicable.

[0150] In a possible implementation, the following scaling mapping relationship may be adopted:

[0151] f(r)=(1+(r-1) 2 *ε)

[0152] Where f(r) is the scaling mapping expression, ε is the degree of eyebrow scaling, and the value range is [0, 1].

[0153] Therefore, for any pixel point (x, y) in the deformation area to be processed, the pixel point position after mapping is (x′, y′), which is expressed as follows:

[0154]

[0155] Generally, eyebrows are long and narrow along the x-axis. Therefore, in addition to the above scaling mapping relationship, asymmetric scaling ratios can also be used for mapping eyebrows. For example, the left and right parts of the eyebrows have different scaling ratios, or the upper and lower parts have different scaling ratios, etc.

[0156] In an embodiment of the present application, since the number of pixels before reduction is greater than the number of pixels after reduction, some pixels cannot be mapped to the pixels after scaling, and thus these pixels are discarded; or, multiple pixels are mapped to the same pixel after scaling, then the pixel after scaling can select the feature mean of these multiple pixels or the feature maximum.

[0157] Step 203: Obtain a target eyebrow template including a virtual makeup effect, and establish a one-to-one pixel mapping relationship between the first actual eyebrow area and the simulated eyebrow area based on multiple first key points and multiple second key points corresponding to the simulated eyebrow area in the eyebrow template.

[0158] In one possible implementation, the target eyebrow template may be obtained by triggering a user selection operation. For example, before capturing a video or image, the user may select a target eyebrow template from among the provided eyebrow templates, or after capturing a video or image, the user may select a corresponding target eyebrow template for the image.

[0159] In another possible implementation, facial recognition may be performed based on the image to be processed provided by the user, and then the most appropriate target eyebrow module may be matched to the face.

[0160] In order to render the eyebrow material to the second image to be processed in real time, a one-to-one pixel mapping relationship can be established between the first actual eyebrow area and the simulated eyebrow area, so that the rendering processing of each pixel point can be performed independently and in parallel, greatly reducing the time complexity and improving the processing speed.

[0161] Specifically, when constructing a pixel mapping relationship, the first actual eyebrow area and the simulated eyebrow area can be gridded in advance to obtain corresponding first grid structures and second grid structures, and then a pixel mapping relationship can be established based on the grid correspondence between the first grid structure and the second grid structure.

[0162] In the first grid structure, the first key point is the vertex of the grid in the first grid structure, and one grid corresponds to a sub-area of ​​the first actual eyebrow area; similarly, in the second grid structure, multiple second key points in the simulated eyebrow area are the vertices of the grid in the second grid structure, and one grid corresponds to a sub-area of ​​the simulated eyebrow area, and the grids in the first grid structure correspond one-to-one to the grids in the second grid structure.

[0163] In a possible implementation, the mesh may be constructed using a triangular mesh-based mesh construction method to perform triangular meshing on the eyebrow region.

[0164] Specifically, the triangular mesh-based mesh construction method can construct a triangular mesh structure based on a given set of key points, using these key points as endpoints to form closed line segments, and construct a triangular mesh structure from these closed line segments, and the triangular mesh structure meets the following conditions:

[0165] (1) Except for the endpoints, the edges in the triangular mesh structure do not contain any points in the point set.

[0166] (2) There are no intersecting edges.

[0167] (3) All faces in the triangular mesh structure are triangular faces, and the union of all triangular faces is the convex hull of the key point set.

[0168] Among them, the more commonly used triangulation method is the Delaunay triangulation algorithm, which is a special triangulation. Among them, the Delaunay edge in the Delaunay algorithm satisfies the empty circle property, that is, the closed line segment e formed by two key points 1 and 2 in the key point set satisfies the empty circle property if there exists a circle passing through points 1 and 2, and the circle does not contain any other points in the key point set. Then the closed line segment e is called a Delaunay edge. Therefore, if a triangulation of the key point set contains only Delaunay edges, then the triangulation is called a Delaunay triangulation.

[0169] Furthermore, the triangular mesh structure obtained by Delaunay triangulation satisfies the following conditions:

[0170] Any triangulation T of a triangular mesh satisfies that T is a Delaunay triangulation of a keypoint set if and only if the interior of the circumcircle of every triangle in T does not contain any point in the keypoint set.

[0171] In the embodiment of the present application, whether it is the key points of the actual eyebrow area in the image to be processed or the key points of the simulated eyebrow area in the eyebrow template, their number is consistent. The only difference is that due to the different eyebrow shapes, the distances between the key points are different, but the relative positional relationship of the key points is also consistent. Therefore, the connection relationship of the grid structure can be constructed in advance based on the standard face, that is, which key points in the finally constructed grid structure are connected to form a subdivision surface. In actual applications, this connection relationship can be directly referenced to quickly perform gridding processing to form a grid structure.

[0172] See also Figure 9 As shown in FIG, a schematic diagram of the corresponding relationship of the constructed grid structure. In the first actual eyebrow area, each first key point constitutes a first grid structure, such as Figure 9 The first key points 1, 2 and 18 shown in the figure form a grid, which corresponds to a sub-region of the first actual eyebrow region, and so on. Similarly, in the simulated eyebrow region, each second key point also forms a second grid structure, and the connection relationship in the second grid structure is consistent with the first grid structure, as shown in FIG. Figure 9The second key points 1, 2 and 18 shown form a grid, which corresponds to a sub-region of the simulated eyebrow region, and the grid formed by the second key points 1, 2 and 18 corresponds to the grid formed by the second key points 1, 2 and 18 in the first grid structure. Figure 9 The dotted arrows shown in the figure represent the corresponding relationship, and the others can be deduced in the same way.

[0173] Then, based on the correspondence between each grid in the first actual eyebrow area and each grid in the simulated eyebrow area, a pixel mapping relationship is constructed for each corresponding grid, for example, Figure 9 The grid formed by the first key points 1, 2 and 18 and the grid formed by the second key points 1, 2 and 18 are shown, and a pixel mapping relationship is established between the two grids.

[0174] In the examples of this application, see Figure 9 As shown, considering that the mesh constructed by key points may not cover all areas of the eyebrow, auxiliary key points can be added. Therefore, when performing meshing processing, each key point and at least one set auxiliary key point can be used as vertices, and meshing processing can be performed based on the set meshing processing method to obtain the corresponding mesh structure. Figure 10 As shown, this is the grid structure diagram after adding auxiliary key points, where points A to K are auxiliary key points. It can be seen that after adding auxiliary key points, all areas of the eyebrow area can be well included, making the makeup effect more realistic.

[0175] Among them, at least one auxiliary key point is a pixel point located outside the first actual eyebrow area, and the grid processing method is determined based on the standard eyebrow area in the standard face image, the standard face image is obtained by averaging various types of face templates, and the face image is in a frontal view state. The set grid processing method is such as the above-mentioned pre-obtained grid connection relationship.

[0176] Step 204: Based on the pixel mapping relationship, extract the rendering materials of each pixel point from the simulated eyebrow area, and perform fusion rendering processing on the corresponding pixel points in the second image to be processed to obtain the target image.

[0177] In an embodiment of the present application, a pixel mapping relationship is established through a grid-based mapping method. After that, the texture mapping technology of computational graphics can be used to sample materials from the eyebrow template, thereby mapping its texture to the second image to be processed, thereby obtaining the target image.

[0178] For example, a texture mapping method based on a triangular mesh can be used to triangulate the simulated eyebrow area in the eyebrow template. That is, the eyebrow key points are used as the vertices of the mesh, and the triangular mesh structure of the eyebrow area is obtained using the Delaunay algorithm. Similarly, for the first actual eyebrow area in the image to be processed, such a triangular mesh structure can also be constructed, so that the topological structure of the triangular mesh can be used to sample from the eyebrow template to obtain the texture mapping of the second image to be processed after the eyebrow template is reduced. This method can use the image processing unit (GPU) to process all pixels in parallel, greatly reducing complexity and improving processing speed.

[0179] The above-mentioned image rendering process of the embodiment of the present application can be applied to various image processing scenarios, such as image rendering in a video stream. The following is an example of image rendering in a video stream. Figure 11 The figure shows a flow chart of image rendering in a video stream.

[0180] Step 1101: Obtain the video stream to be processed.

[0181] Specifically, the video stream to be processed can be selected by the user from a local photo album, or can be shot by the user in real time.

[0182] Step 1102: Determine whether the current video frame contains a human face.

[0183] Step 1103: If the result of step 1102 is yes, the current video frame is determined as the image to be processed.

[0184] Step 1104: Perform image rendering processing on the image to be processed to obtain a corresponding target image.

[0185] Step 1105: Based on the target images corresponding to the images to be processed, a target video stream containing the virtual makeup effect is obtained.

[0186] Step 1106: If the result of step 1102 is no, jump to the next video frame and continue processing.

[0187] In the embodiment of the present application, the rendering process of each image to be processed can be performed sequentially according to a queue, or the rendering process of multiple images to be processed can be performed in parallel.

[0188] In a possible implementation, the above-mentioned video stream processing process may be performed locally by the terminal device.

[0189] In another possible implementation, the aforementioned video stream processing can be performed by the terminal device uploading the video stream to a backend server, which then provides background processing services. For example, a user may select a virtual makeup effect on a camera application on the terminal device and begin shooting a video using that application. The video stream is then uploaded to the backend server, which processes each video frame using the aforementioned process and returns the target video stream to the terminal device. The terminal device then displays the target video stream in real time, allowing the user to view the target video stream with the virtual makeup effect applied.

[0190] To sum up, the embodiment of the present application adopts an augmented reality method to liquefy and deform the eyebrow area, locally scale the eyebrows, and then fit different eyebrows through augmented reality to obtain diversified eyebrow shapes and natural makeup effects. The eyebrow reduction scheme prevents the eyebrow template from being obviously mixed and superimposed with the original eyebrow, making the eyebrow makeup effect more natural. In addition, by utilizing graphics texture fitting, the eyebrow template is texture sampled, so that the eyebrows can be made up in real time.

[0191] See Figure 12 Based on the same inventive concept, the embodiment of the present application further provides an image rendering processing device 120, which includes:

[0192] The face recognition unit 1201 is configured to perform facial key point recognition on a first image to be processed containing a face, and identify a plurality of first key points corresponding to a first actual eyebrow region of the face from the first image to be processed;

[0193] The scaling mapping unit 1202 is configured to determine, in the first image to be processed, a deformed region to be processed including the first actual eyebrow region based on the plurality of first key points, and perform scaling mapping on each pixel in the deformed region to be processed to obtain a second image to be processed including the scaled second actual eyebrow region;

[0194] A grid construction unit 1203 is configured to obtain a target eyebrow template including a virtual makeup effect, and establish a one-to-one pixel mapping relationship between the first actual eyebrow region and the simulated eyebrow region based on the plurality of first key points and the plurality of second key points corresponding to the simulated eyebrow region in the eyebrow template;

[0195] The texture fitting unit 1204 is used to extract rendering materials of each pixel point from the simulated eyebrow area based on the pixel mapping relationship, and perform fusion rendering processing on the corresponding pixel points in the second image to be processed to obtain a target image.

[0196] Optionally, the scaling mapping unit 1202 is specifically configured to:

[0197] Determining reference points of the deformation region to be processed in the first image to be processed based on coordinate values ​​of the plurality of first key points in the constructed reference coordinate system;

[0198] Based on the set shape model, a deformation area to be processed is determined in the first image to be processed with the reference point as the center.

[0199] Optionally, the scaling mapping unit 1202 is specifically configured to:

[0200] The pixel point corresponding to the average of the coordinate values ​​corresponding to the plurality of first key points is determined as the reference point.

[0201] Optionally, the scaling mapping unit 1202 is specifically configured to:

[0202] Determine the distances between the reference point and each first key point, determine a first distance having the largest value from the obtained distances, and determine the semi-major axis based on the first distances;

[0203] Selecting a second distance with a larger value from the distances between the reference point and the two nearest first key points, and determining the semi-minor axis based on the second distance;

[0204] The elliptical area surrounded by the semi-major axis and the semi-minor axis with the reference point as the center is determined as the deformation area to be processed.

[0205] Optionally, the scaling mapping unit 1202 is specifically configured to:

[0206] Determine an angle between a line connecting the reference point and a first key point corresponding to the first distance and a set coordinate axis in the reference coordinate system;

[0207] For each pixel in the image to be processed, perform the following operations:

[0208] For a pixel point, after rotating the pixel point by the angle value with the reference point as the center, the distance between the position of the rotated pixel point and the reference point is determined;

[0209] If the distance between the position of a pixel point after rotation and the reference point is less than a set distance threshold, the pixel point is determined to be a pixel point located in the deformation area to be processed.

[0210] Optionally, the scaling mapping unit 1202 is specifically configured to:

[0211] Determine the distances between the reference point and each first key point, and determine the distance with the largest value from the obtained distances;

[0212] The circular area with the reference point as the center and the largest distance as the radius is determined as the deformation area to be processed.

[0213] Optionally, the scaling mapping unit 1202 is specifically configured to:

[0214] For each pixel in the deformed area to be processed, perform the following operations to obtain a second image to be processed:

[0215] For each pixel point, based on the coordinate value of the pixel point and a set scaling mapping relationship, determining a scaled coordinate value of the pixel point; wherein, in the set scaling mapping relationship, the scaling ratio is negatively correlated with the distance between the reference points of the deformation area to be processed;

[0216] The pixel feature value of a pixel point is used to update the pixel point at the location of the scaled coordinate value.

[0217] Optionally, the grid construction unit 1203 is specifically configured to:

[0218] Based on the plurality of first key points, meshing is performed on the first actual eyebrow region to obtain a first mesh structure; wherein the plurality of first key points are vertices of meshes in the first mesh structure, and one mesh corresponds to a sub-region of the first actual eyebrow region; and

[0219] Based on the plurality of second key points, meshing is performed on the simulated eyebrow region to obtain a second mesh structure; wherein the plurality of second key points are vertices of meshes in the second mesh structure, and each mesh corresponds to a sub-region of the simulated eyebrow region, and the meshes in the first mesh structure correspond one-to-one to the meshes in the second mesh structure;

[0220] A pixel mapping relationship is established based on the grid correspondence relationship between the first grid structure and the second grid structure.

[0221] Optionally, the grid construction unit 1203 is specifically configured to:

[0222] Taking the plurality of first key points and at least one set auxiliary key point as vertices, and based on a set gridding processing method, performing gridding processing on the first actual eyebrow region to obtain a first grid structure;

[0223] The gridding processing method is determined based on the standard eyebrow area in the standard face image. The standard face image is a face image in a frontal view state obtained by averaging various types of face templates.

[0224] Optionally, the apparatus further includes a video stream processing unit 1205, configured to:

[0225] A video stream to be processed is obtained, and an image containing a human face is selected from the video stream to be processed as an image to be processed; and a target video stream containing a virtual makeup effect is obtained based on target images corresponding to each image to be processed.

[0226] The device can be used to perform Figures 2 to 11 The method shown in the embodiment shown, therefore, for the functions that can be realized by each functional module of the device, please refer to Figures 2 to 11 The description of the embodiment shown in FIG. 1 is omitted. The video stream processing unit 1205 is an optional functional module. Figure 12 It is shown with a dotted line.

[0227] Based on the same technical concept as the above method embodiment, an embodiment of the present application further provides a computer device 130 that may include a memory 1301 and a processor 1302 .

[0228] The memory 1301 is used to store computer programs executed by the processor 1302. The memory 1301 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function, etc.; the data storage area may store data created according to the use of the computer device, etc. The processor 1302 may be a central processing unit (CPU), or a digital processing unit, etc. The specific connection medium between the above-mentioned memory 1301 and the processor 1302 is not limited in the embodiment of the present application. The embodiment of the present application is Figure 13 In the embodiment, the memory 1301 and the processor 1302 are connected via a bus 1303. The bus 1303 is connected to the processor 1302 via a bus 1303. Figure 13 The bus 1303 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 13 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0229] Memory 1301 may be a volatile memory, such as random-access memory (RAM); a non-volatile memory, such as read-only memory, flash memory, a hard disk drive (HDD), or a solid-state drive (SSD); or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto. Memory 1301 may be a combination of the above memories.

[0230] The processor 1302 is configured to execute the following when calling the computer program stored in the memory 1301: Figures 2 to 11 The method executed by the device in the embodiment shown.

[0231] Refer to the following Figure 14 140 according to this embodiment of the present application. Figure 14 The computing device 140 is merely an example and should not limit the functionality and scope of use of the embodiments of the present application.

[0232] like Figure 14 As shown, computing device 140 is implemented as a general-purpose computing device. Components of computing device 140 may include, but are not limited to, at least one processing unit 1401, at least one storage unit 1402, and a bus 1403 connecting various system components (including storage unit 1402 and processing unit 1401).

[0233] Bus 1403 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, and a processor or local bus using any of a variety of bus architectures.

[0234] The storage unit 1402 may include a readable medium in the form of a volatile memory, such as a random access memory (RAM) 14021 and / or a cache memory unit 14022 , and may further include a read-only memory (ROM) 14023 .

[0235] The storage unit 1402 may also include a program / utility 14025 having a set (at least one) of program modules 14024, such program modules 14024 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0236] Computing device 140 may also communicate with one or more external devices 1404 (e.g., keyboards, pointing devices, etc.), as well as with one or more devices that enable a user to communicate with other computing devices and / or any device that enables computing device 140 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may occur via input / output (I / O) interface 1405. Furthermore, computing device 140 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks such as the Internet) via network adapter 1406. As shown, network adapter 1406 communicates with other modules of computing device 140 via bus 1403. It should be understood that, although not shown, other hardware and / or software modules may be used in conjunction with computing device 140, including but not limited to microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0237] In some possible implementations, various aspects of the method provided in the present application may also be implemented in the form of a program product, which includes program code. When the program product is run on a computer device, the program code is used to enable the computer device to perform the steps of the method according to various exemplary embodiments of the present application described above in this specification. For example, the computer device may perform the following steps: Figures 2 to 11 The method executed by the device in the embodiment shown.

[0238] The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0239] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present application.

[0240] Obviously, those skilled in the art may make various changes and modifications to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalents, this application is intended to include these modifications and variations.

Claims

1. An image rendering processing method, characterized in that: The method comprises: Performing facial key point recognition on a first image to be processed containing a human face, and identifying a plurality of first key points corresponding to a first actual eyebrow region of the human face from the first image to be processed; Based on the multiple first key points, determining, in the first image to be processed, a deformed region to be processed that includes the first actual eyebrow region, and performing scaling mapping processing on each pixel point in the deformed region to be processed to obtain a second image to be processed that includes a scaled second actual eyebrow region; Obtaining a target eyebrow template including a virtual makeup effect, and establishing a one-to-one pixel mapping relationship between the first actual eyebrow region and the simulated eyebrow region based on the plurality of first key points and a plurality of second key points corresponding to the simulated eyebrow region in the eyebrow template; Based on the pixel mapping relationship, rendering materials of each pixel point are extracted from the simulated eyebrow area, and fusion rendering processing is performed on the corresponding pixel points in the second image to be processed to obtain a target image.

2. The method according to claim 1, wherein Determining, based on the plurality of first key points, in the first image to be processed, a deformed region to be processed that includes the first actual eyebrow region, comprises: Based on the coordinate values ​​of the plurality of first key points in the constructed reference coordinate system, determining, in the first image to be processed, reference points of the deformation area to be processed; Based on the set shape model, the deformation area to be processed is determined in the first image to be processed with the reference point as the center.

3. The method according to claim 2, wherein Determining, in the first image to be processed, reference points of the deformed area to be processed based on coordinate values ​​of the plurality of first key points in the constructed reference coordinate system, includes: The pixel point corresponding to the average of the coordinate values ​​corresponding to the plurality of first key points is determined as the reference point.

4. The method according to claim 2, wherein If the shape model is an ellipse model, determining the deformation area to be processed in the first image to be processed based on the set shape model and taking the reference point as the center includes: Determining the distances between the reference point and each first key point, determining a first distance having the largest value from the obtained distances, and determining a semi-major axis based on the first distances; Selecting a second distance with a larger value from the distances between the reference point and the two nearest first key points, and determining the semi-minor axis based on the second distance; An elliptical area centered at the reference point and enclosed by the semi-major axis and the semi-minor axis is determined as the deformation area to be processed.

5. The method according to claim 4, wherein Before performing scaling mapping processing on each pixel point in the to-be-processed deformed area to obtain a second to-be-processed image including a scaled second actual eyebrow area, the method further includes: Determine an angle between a line connecting the reference point and a first key point corresponding to the first distance and a set coordinate axis in the reference coordinate system; For each pixel in the image to be processed, perform the following operations: For a pixel point, after rotating the pixel point by the angle value with the reference point as the center, determine the distance between the position of the rotated pixel point and the reference point; If the distance between the position of the rotated pixel point and the reference point is less than a set distance threshold, the pixel point is determined to be a pixel point located in the deformation area to be processed.

6. The method according to claim 2, wherein If the shape model is a circular model, determining the deformation area to be processed in the first image to be processed based on the set shape model and taking the reference point as the center includes: Determining the distances between the reference point and each first key point, and determining the distance with the largest value from the obtained distances; A circular area with the reference point as the center and the maximum distance as the radius is determined as the deformation area to be processed.

7. The method according to any one of claims 1 to 6, wherein: Performing scaling mapping processing on each pixel point in the to-be-processed deformed area to obtain a second to-be-processed image containing a scaled second actual eyebrow area, comprising: For each pixel in the deformed area to be processed, perform the following operations to obtain the second image to be processed: For a pixel point, determining a scaled coordinate value of the pixel point based on the coordinate value of the pixel point and a set scaling mapping relationship; wherein, in the set scaling mapping relationship, the scaling ratio is negatively correlated with the distance between the reference points of the deformation area to be processed; The pixel point at the location of the scaled coordinate value is updated using the pixel feature value of the one pixel point.

8. The method according to any one of claims 1 to 6, wherein: Establishing a one-to-one pixel mapping relationship between the first actual eyebrow area and the simulated eyebrow area based on the multiple first key points and the multiple second key points corresponding to the simulated eyebrow area in the eyebrow template includes: Based on the multiple first key points, meshing the first actual eyebrow region to obtain a first mesh structure; wherein the multiple first key points are vertices of meshes in the first mesh structure, and one mesh corresponds to a sub-region of the first actual eyebrow region; and Based on the plurality of second key points, meshing the simulated eyebrow region is performed to obtain a second mesh structure; wherein the plurality of second key points are vertices of meshes in the second mesh structure, and one mesh corresponds to a sub-region of the simulated eyebrow region, and the meshes in the first mesh structure correspond one-to-one to the meshes in the second mesh structure; The pixel mapping relationship is established based on the grid correspondence relationship between the first grid structure and the second grid structure.

9. The method according to claim 8, wherein Based on the plurality of first key points, meshing the first actual eyebrow region to obtain a first mesh structure includes: Taking the multiple first key points and the at least one set auxiliary key point as vertices, and based on a set gridding processing method, performing gridding processing on the first actual eyebrow area to obtain the first grid structure; The gridding processing method is determined based on a standard eyebrow region in a standard face image, and the standard face image is a face image in a frontal view state obtained by averaging various types of face templates.

10. The method according to any one of claims 1 to 6, wherein: Before performing facial key point recognition on the image to be processed containing the human face and identifying a plurality of first key points corresponding to the actual eyebrow region of the human face from the image to be processed, the method further includes: Acquire a video stream to be processed, and select an image containing a face from the video stream to be processed as the image to be processed; Then, after extracting rendering materials of each pixel point from the eyebrow template based on the pixel mapping relationship and performing fusion rendering processing on the corresponding pixel points in the eyebrow-reduced image to be processed to obtain the target image, the method further includes: Based on the target images corresponding to the images to be processed, a target video stream containing the virtual makeup effect is obtained.

11. An image rendering processing device, characterized in that: The device comprises: a face recognition unit, configured to perform facial key point recognition on a first image to be processed containing a face, and identify a plurality of first key points corresponding to a first actual eyebrow region of the face from the first image to be processed; a scaling mapping unit configured to determine, in the first image to be processed, a deformed region to be processed that includes the first actual eyebrow region based on the plurality of first key points, and perform scaling mapping processing on each pixel in the deformed region to be processed to obtain a second image to be processed that includes a scaled second actual eyebrow region; a grid construction unit, configured to obtain a target eyebrow template including a virtual makeup effect, and establish a one-to-one pixel mapping relationship between the first actual eyebrow region and the simulated eyebrow region based on the plurality of first key points and a plurality of second key points corresponding to the simulated eyebrow region in the eyebrow template; The texture fitting unit is used to extract rendering materials of each pixel point from the simulated eyebrow area based on the pixel mapping relationship, and perform fusion rendering processing on the corresponding pixel points in the second image to be processed to obtain a target image.

12. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 10 are implemented.

13. A computer storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the steps of the method according to any one of claims 1 to 10 are implemented.

Images