Face image processing method, device, equipment and computer-readable storage medium

Abstract

The application provides a face image processing method, device and equipment and a computer readable storage medium. The method comprises: performing face key point detection on a to-be-processed face image to obtain a face key point set; determining a first forehead feature point based on a first feature point representing a nasal root and a second feature point representing a lower jaw in the face key point set; determining a second forehead feature point and a third forehead feature point based on the first forehead feature point and the first feature point, and in combination with a third feature point representing the outer side of a left face contour and a fourth feature point representing the outer side of a right face contour, respectively; performing interpolation fitting based on the first forehead feature point, the second forehead feature point and the third forehead feature point to obtain a forehead feature point set; and performing image processing on the to-be-processed face image based on the forehead feature point set to obtain an image processing result. Through the application, the efficiency and accuracy of face image processing can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to artificial intelligence technology, and in particular to a face image processing method and device, equipment and a computer readable storage medium. BACKGROUND

[0002] At present, through a face feature point model based on deep machine learning, most key parts in a to-be-processed face image, such as nose, cheek, eye and the like, can be obtained. However, for the forehead region, due to the influence of factors such as hairstyle and hairline shape, the semantic information of the forehead feature point output by the face feature point model is not clear, so the current neural network output face key point set often does not contain the feature point of the forehead region. In order to obtain the feature point of the forehead region, the current main method is to use a modeling method based on a geometric model, to splice multiple polynomials or elliptical shapes to fit the face shape. This method is not only computationally complex, but also has low adaptability to various different face shapes, thereby reducing the efficiency and accuracy of face image processing. SUMMARY

[0003] The embodiments of the present application provide a face image processing method, device and computer readable storage medium, which can improve the efficiency and accuracy of face image processing.

[0004] The technical scheme of the embodiments of the present application is implemented as follows:

[0005] The embodiments of the present application provide a face image processing method, comprising:

[0006] performing face key point detection on a to-be-processed face image to obtain a face key point set;

[0007] determining a first forehead feature point based on a first feature point representing a nasal root and a second feature point representing a lower jaw in the face key point set; the first forehead feature point represents the highest point of the forehead on the external contour of the face;

[0008] determining a second forehead feature point and a third forehead feature point based on the first forehead feature point and the first feature point, in combination with a third feature point representing the left face contour outside and a fourth feature point representing the right face contour outside, respectively; the third feature point and the fourth feature point belong to the face key point set;

[0009] performing interpolation fitting based on the first forehead feature point, the second forehead feature point and the third forehead feature point to obtain a forehead feature point set; the forehead feature point set represents the forehead contour corresponding to the to-be-processed face;

[0010] performing image processing on the to-be-processed face image based on the forehead feature point set to obtain an image processing result.

[0011] The embodiment of the present application provides a face image processing device, comprising:

[0012] A face key point detection model is configured to perform face key point detection on a face image to be processed to obtain a face key point set.

[0013] A determination module is configured to determine a first forehead feature point based on a first feature point representing a nasal root and a second feature point representing a lower jaw in the face key point set; the first forehead feature point represents a highest point of a forehead on an external contour of a face; and determine a second forehead feature point and a third forehead feature point based on the first forehead feature point and the first feature point, respectively combined with a third feature point representing a left face contour outside and a fourth feature point representing a right face contour outside.

[0014] An interpolation fitting module is configured to perform interpolation fitting based on the first forehead feature point, the second forehead feature point and the third forehead feature point to obtain a forehead feature point set; the forehead feature point set represents a forehead contour corresponding to the face image to be processed.

[0015] A processing module is configured to perform image processing on the face image to be processed based on the forehead feature point set to obtain an image processing result.

[0016] In the device, the determination module is further configured to determine the first forehead feature point on a vector pointing from the second feature point to the first feature point based on a distance between the first feature point and the second feature point.

[0017] In the device, the determination module is further configured to multiply a preset adjustment factor by a horizontal coordinate and a vertical coordinate of the first feature point respectively to obtain a first horizontal product and a first vertical product; the preset adjustment factor is a value greater than 1; calculate a difference between a preset threshold value and the preset adjustment factor, and multiply the difference by the horizontal coordinate and the vertical coordinate of the second feature point respectively to obtain a second horizontal product and a second vertical product; and take a sum of the first horizontal product and the second horizontal product as the horizontal coordinate of the first forehead feature point, and take a sum of the first vertical product and the second vertical product as the vertical coordinate of the first forehead feature point, thereby determining the first forehead feature point.

[0018] In the device, the determination module is further configured to calculate a first distance between the second feature point and the first feature point, and calculate a second distance between a preset fixed distance point and the first feature point; the distance between the preset fixed distance point and the first feature point changes less than a preset change threshold in the case of a change in the angle of the face; the preset fixed distance point belongs to the set of face key points; and the preset adjustment factor is obtained based on the ratio of the first distance to the second distance.

[0019] In the device, the determination module is further configured to calculate a first feature vector and a first length of the first feature point pointing to the first forehead feature point, a second feature vector and a second length of the first feature point pointing to the third feature point, and a third feature vector and a third length of the first feature point pointing to the fourth feature point; calculate an intermediate included angle between the first feature vector and the second feature vector to determine a first intermediate feature vector; calculate a first average value of the first length and the second length; determine the second forehead feature point according to the first intermediate feature vector and the first average value; calculate an intermediate included angle between the first feature vector and the third feature vector to determine a second intermediate feature vector; calculate a second average value of the first length and the third length; and determine the third forehead feature point according to the second intermediate feature vector and the second average value.

[0020] In the device, the interpolation fitting module is further configured to obtain a forehead curve constraint corresponding to the face to be processed according to the third feature point, the second forehead feature point, the first forehead feature point, the third forehead feature point, and the fourth feature point; and perform interpolation fitting between the third feature point, the second forehead feature point, the first forehead feature point, the third forehead feature point, and the fourth feature point based on the forehead curve constraint to obtain the forehead feature point set containing the first forehead feature point, the second forehead feature point, and the third forehead feature point.

[0021] In the device, the processing module is further configured to divide the to-be-processed face image into a plurality of real face regions according to the set of face key points and the set of forehead feature points; obtain a preset special effect face corresponding to the to-be-processed face image; the preset special effect face is a face template containing a preset special effect image; the preset special effect face contains a plurality of preset face regions obtained by performing the same feature point calculation and division processing on the face template in advance; obtain a special effect pixel corresponding to each face pixel in the to-be-processed face image in the preset special effect face according to the correspondence between the plurality of real face regions and the plurality of preset face regions; and perform pixel fusion on the each face pixel and the corresponding special effect pixel to obtain an image processing result in which the to-be-processed face image and the preset special effect image are superimposed.

[0022] In the device, the processing module is further configured to perform triangular mesh division on the set of face key points and the set of forehead feature points by taking each feature point in the set of face key points and the set of forehead feature points as a vertex and using a mesh division algorithm to obtain the plurality of real face regions formed by triangular meshes.

[0023] In the device, the processing module is further configured to, for each face pixel, perform weighted calculation according to the vertex position of a target real face region in which the each face pixel is located to obtain a relative position of the each face pixel in the target real face region; determine a target preset face region corresponding to the target real face region according to the correspondence between the plurality of real face regions and the plurality of preset face regions; and determine a pixel corresponding to the relative position in the target preset face region as a special effect pixel corresponding to the each face pixel according to the vertex position of the target preset face region.

[0024] In the device, the processing module is further configured to perform parallel fusion processing on the each face pixel and the corresponding material pixel by using a graphics processing module to obtain the image processing result.

[0025] In the device, the processing module is further configured to perform chroma fusion on the each face pixel and the corresponding material pixel to obtain an intermediate chroma; perform fusion intensity adjustment on the chroma of the each face pixel and the intermediate chroma by using a preset fusion intensity factor to obtain a first adjustment result and a second adjustment result; and combine the first adjustment result and the second adjustment result to obtain an image fusion result corresponding to the each face pixel as the image processing result.

[0026] In the device, the processing module is further configured to perform at least one of face segmentation, face alignment, face recognition, and face synthesis on the to-be-processed face image based on the set of forehead feature points to obtain the image processing result.

[0027] An electronic device is provided in an embodiment of the present application, comprising:

[0028] a memory configured to store executable instructions;

[0029] a processor configured to execute the executable instructions stored in the memory to implement a method provided in an embodiment of the present application.

[0030] A computer readable storage medium is provided in an embodiment of the present application, storing executable instructions configured to cause a processor to implement a method provided in an embodiment of the present application when executed.

[0031] An embodiment of the present application has the following beneficial effects:

[0032] An embodiment of the present application can preliminarily locate a first forehead feature point, a second forehead feature point and a third forehead feature point on a forehead contour according to the key points in the forehead key point set, such as feature points representing a nose root, a left face contour outside and a right face contour outside; and on this basis, more forehead feature points are calculated by using an adaptive interpolation method, so as to obtain a forehead feature point set corresponding to a smoother forehead contour. The calculation process of the embodiment of the present application is faster, and robust forehead feature points can be determined by the adaptive interpolation method for various different face shapes, thereby improving the efficiency and accuracy of forehead feature point recognition, and further improving the efficiency and accuracy of face image processing based on the forehead feature point set. BRIEF DESCRIPTION OF DRAWINGS

[0033] Figure 1 is an optional structural schematic diagram of a face image processing system architecture provided in an embodiment of the present application;

[0034] Figure 2 is an optional structural schematic diagram of a face image processing apparatus provided in an embodiment of the present application;

[0035] Figure 3 is an optional flow schematic diagram of a face image processing method provided in an embodiment of the present application;

[0036] Figure 4 is an optional effect schematic diagram of a forehead key point set provided in an embodiment of the present application;

[0037] Figure 5 is an optional effect schematic diagram of a first forehead feature point provided in an embodiment of the present application;

[0038] Figure 6 is an optional effect schematic diagram of a second forehead feature point provided in an embodiment of the present application;

[0039] Figure 7is an optional effect schematic diagram of a third forehead feature point provided by an embodiment of the present application;

[0040] Figure 8 is an optional effect schematic diagram of a forehead feature point set provided by an embodiment of the present application;

[0041] Figure 9 is an optional flow schematic diagram of a face image processing method provided by an embodiment of the present application;

[0042] Figure 10 is an optional module structure schematic diagram of a face image processing device applied to an online makeup function provided by an embodiment of the present application;

[0043] Figure 11 is an optional flow schematic diagram of a face image processing method provided by an embodiment of the present application. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be described in further detail below with reference to the drawings, and the described embodiments should not be regarded as limiting the present application, and all other embodiments obtained by those of ordinary skill in the art without making creative efforts fall within the scope of protection of the present application.

[0045] In the following description, “some embodiments” are described, which describe a subset of all possible embodiments, but it can be understood that “some embodiments” can be the same subset or different subsets of all possible embodiments, and can be combined with each other without conflict.

[0046] In the following description, the terms “first\second\third” are only to distinguish similar objects, and do not represent a specific order of the objects, and it can be understood that “first\second\third” can be interchanged in a specific order or sequence as allowed, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0047] The related data collection and processing in the embodiments of the present application should strictly comply with the requirements of relevant laws and regulations, obtain the informed consent or separate consent of the personal information subject, and within the scope of authorization of laws and regulations and the personal information subject, carry out subsequent data use and processing.

[0048] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0049] Before the embodiments of the present application are described in further detail, the terms and names involved in the embodiments of the present application are explained, and the terms and names involved in the embodiments of the present application are applicable to the following explanations.

[0050] 1) Artificial Intelligence (AI) is the theory, method, technology and application system of using digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology of computer science, which tries to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that machines have the functions of perception, reasoning and decision-making.

[0051] Artificial intelligence technology is a comprehensive discipline, involving a wide range of fields, both hardware and software technologies. Artificial intelligence basic technologies generally include technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction system, mechatronics, etc. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning, etc.

[0052] 2) Computer Vision (CV): Computer vision is a science that studies how to make machines "see". More specifically, it refers to using cameras and computers to replace human eyes to identify, track and measure targets, and further process images so that the computer processing becomes more suitable for human eye observation or transmission to instrument detection. As a scientific discipline, computer vision researches related theories and technologies, trying to establish artificial intelligence systems that can obtain information from images or multidimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping, etc. It also includes common face recognition, fingerprint recognition and other biometric identification technologies.

[0053] Face key point detection is an important basic link in face recognition tasks, and accurate detection of face key points plays a key role in many real-world applications and research topics, such as face pose recognition and correction, expression recognition, mouth shape recognition, etc. Therefore, how to obtain high-precision face key points has always been a popular research problem in the fields of computer vision and image processing. Influenced by factors such as face pose and occlusion, the research of face key point detection is also challenging.

[0054] 3) Machine Learning (ML) is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithmic complexity theory, and other disciplines. It is a specialized study of how computers simulate or implement human learning behavior to acquire new knowledge or skills, reorganize existing knowledge structure, and continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental approach to making computers intelligent. Its applications span various fields of artificial intelligence. Machine learning and deep learning generally include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and rule-based learning.

[0055] With the research and progress of artificial intelligence technology, artificial intelligence technology is being researched and applied in many fields, such as common smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned vehicles, autonomous vehicles, drones, robots, smart medical care, smart customer service, etc. With the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0056] The scheme provided by the embodiments of the present application relates to computer vision and other technologies of artificial intelligence, which is specifically explained as follows:

[0057] With the advent of the video era, more and more users will make videos and express content through short videos and live streaming. When recording or live streaming videos, a good makeup can effectively modify the face shape, increase the three-dimensional effect of the face, improve the personal temperament, and enhance the user's self-confidence in their own image, and attract more audience's attention and enthusiasm. However, a good makeup often takes a long time and requires various makeup accessories, and also requires the user to have certain makeup experience to make targeted makeup modifications (including concealer, foundation, setting, eyebrow makeup, eye makeup, and lip makeup) according to their own skin and features. This is a high threshold for a large number of user groups. The method used by the automatic makeup technology in the related art mainly performs face recognition positioning on the image frame to be processed, then maps and changes the makeup texture pattern of the virtual makeup, obtains the deformed texture, and finally fuses the deformed texture and the face to be processed, thereby obtaining the effect of automatically superimposing the virtual makeup on the face to be processed in the image frame. The face recognition positioning in the above process of the related art is mainly based on the following two methods:

[0058] I. Through a face analysis model, the face is segmented to obtain each part of the face, so that the makeup texture can be subjected to affine transformation, and then the mapping relationship between the face to be processed and the makeup texture is obtained;

[0059] II. Through the face key point model, face key points of the face to be processed can be obtained, and feature points of the makeup texture are also marked. The mapping relationship between the face to be processed and the makeup texture can also be obtained through the transformation relationship of the feature points.

[0060] For the first method, the face recognition and positioning method using face segmentation is time-consuming and difficult to be directly applied to scenarios with high real-time requirements such as video call and video live broadcast. Moreover, the segmentation result is prone to have a relatively large segmentation error at the occlusion, thereby causing the virtual makeup to be not naturally fitted. In addition, the face image processing accuracy is low.

[0061] For the second method, the real face is often affected by factors such as hairstyle and hairline shape, which causes the face key point model in the related art to not clearly recognize the semantic information of the forehead point. The output face key points do not contain feature points of the forehead region, and thus cannot be applied to some virtual makeup that needs to be applied to the forehead part, such as the Wu Meiniang makeup, highlight makeup, and some sticker special effects. The related art has a modeling method based on a geometric model. This method assumes that the face shape can be fitted by multiple polynomials or an elliptical shape. However, this method is complex in calculation, increases the calculation amount and processing time of the network model, and reduces the face image processing efficiency. Moreover, this method cannot well adapt to various face shapes and various face angle changes (such as head raising, head lowering, and side face), thereby causing the forehead key point position to be inaccurate, resulting in a virtual makeup superposition effect that is not well fitted, and reducing the face image processing accuracy.

[0062] In addition, the two methods described above both obtain the affine transformation relationship between the face to be processed and the makeup texture through a small number of feature points, obtain an affine mapping matrix, and perform a certain distance translation, a certain angle rotation, or a certain scale zoom on the face to be processed to obtain the mapping relationship between the face to be processed and the makeup texture. The mapping relationship obtained from the small number of feature points in the related art has low accuracy, which easily causes the forehead region to be not naturally fitted, and also reduces the face image processing accuracy.

[0063] The embodiments of the present application provide a face image processing method, device, equipment, and computer readable storage medium, which can improve the efficiency and accuracy of face image processing. The following describes an exemplary application of the electronic device provided by the embodiments of the present application. The electronic device provided by the embodiments of the present application can be implemented as a notebook computer, a tablet computer, a desktop computer, a set-top box, a mobile device (for example, a mobile phone, a portable music player, a personal digital assistant, a dedicated message device, a portable game device), and various types of user terminals. The electronic device can also be implemented as a server. The following describes an exemplary application of the electronic device implemented as a server.

[0064] Referring toFigure 1 , Figure 1 is an optional architecture schematic diagram of the face image processing system 100 provided by the embodiments of the present application, in order to realize support for a face image processing application, such as an online makeup application, a video live streaming beautifying application, etc., the terminal 400 (exemplarily shows the terminal 400-1 and the terminal 400-2) connects the server 200 through the network 300, and the network 300 can be a wide area network or a local area network, or a combination of the two.

[0065] The terminal 400-1 is configured to receive the operation of the anchor user through the graphical interface 410-1, and collect the face image of the anchor user as a to-be-processed face image based on the operation of the anchor user. The terminal 400-1 is further configured to receive the image processing mode specified by the anchor user through the graphical interface 410-1, such as selecting a virtual makeup from a virtual makeup list displayed on the graphical interface 410-1 and applying the virtual makeup to the current to-be-processed face image, and sending the to-be-processed face image and the identifier of the virtual makeup specified by the anchor user to the server 200.

[0066] The server 200 is configured to perform face key point detection on the to-be-processed face image to obtain a face key point set; determine a first forehead feature point based on a first feature point representing a nasal root and a second feature point representing a lower jaw in the face key point set; the first forehead feature point represents the highest point of the forehead on the external contour of the face; determine a second forehead feature point and a third forehead feature point based on the first forehead feature point, the first feature point, a third feature point representing the left side of the face contour, and a fourth feature point representing the right side of the face contour; the third feature point and the fourth feature point belong to the face key point set; perform interpolation fitting based on the first forehead feature point, the second forehead feature point, and the third forehead feature point to obtain a forehead feature point set; the forehead feature point set represents the forehead contour corresponding to the to-be-processed face image; obtain face feature points that can mark each part in the to-be-processed face image based on the forehead feature point set and the face key point set; the server 200 can obtain the image of the preset virtual makeup from the database 500 according to the identifier of the virtual makeup specified by the first user, and then fuse the preset virtual makeup with the to-be-processed face image according to the face feature points of each part in the to-be-processed face image to obtain the beautifying makeup effect of the to-be-processed face image as the image processing result, thereby realizing the image processing process of the to-be-processed face image. The server 200 further sends the image processing result, i.e., the beautifying makeup effect of the to-be-processed face image, to the terminal 400-1 and the terminal 400-2 through the network 300, and synchronously displays the image processing result on the graphical interface 410-1 of the terminal 400-1 and the graphical interface 410-2 of the terminal 400-2 for the anchor user and the audience user of the terminal 400-2.

[0067] In some embodiments, the server 200 can be a standalone physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDNs, and basic cloud computing services such as big data and artificial intelligence platforms. The terminal 400 can be a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, and the like, but is not limited thereto. The terminal and the server can be connected directly or indirectly through wired or wireless communication, and the present application is not limited in this embodiment.

[0068] It should be noted that when the electronic device is implemented as a terminal, the terminal can collect the face image to be processed, and execute the face image processing method provided in the present application locally to obtain the image processing result.

[0069] Referring to Figure 2 , Figure 2 is a structural schematic diagram of the server 200 provided in the present application, Figure 2 The server 200 shown in FIG. 1 includes at least one processor 210, a memory 250, at least one network interface 220, and a user interface 230. The various components in the server 200 are coupled together by a bus system 240. It can be understood that the bus system 240 is used to realize the connection and communication between the components. In addition to the data bus, the bus system 240 also includes a power bus, a control bus, and a status signal bus. However, for the purpose of clear illustration, all kinds of buses are marked as the bus system 240 in Figure 2 .

[0070] The processor 210 can be an integrated circuit chip with signal processing capability, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0071] The user interface 230 includes one or more output devices 231 that enable the presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 230 also includes one or more input devices 232, including user interface components that facilitate user input, such as a keyboard, a mouse, a microphone, a touch screen display, a camera, other input buttons and controls.

[0072] The memory 250 can be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard drives, optical drives, and the like. The memory 250 optionally includes one or more storage devices remotely located from the processor 210.

[0073] The memory 250 includes volatile memory or nonvolatile memory, and can also include both volatile and nonvolatile memory. Nonvolatile memory can be read only memory (ROM), volatile memory can be random access memory (RAM). The memory 250 described in the embodiments of the present application is intended to include any suitable type of memory.

[0074] In some embodiments, the memory 250 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, which are exemplarily illustrated below.

[0075] The operating system 251 includes system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, and the like, for implementing various basic services and processing hardware-based tasks;

[0076] The network communication module 252 is used to reach other computing devices via one or more (wired or wireless) network interfaces 220, exemplary network interfaces 220 including Bluetooth, wireless compatibility certification (WiFi), and universal serial bus (USB), and the like;

[0077] The presentation module 253 is used to enable the presentation of information via one or more output devices 231 associated with the user interface 230 (e.g., a display screen, a speaker, and the like) (e.g., a user interface for operating peripheral devices and displaying content and information);

[0078] The input processing module 254 is used to detect and translate one or more user inputs or interactions from one or more input devices 232.

[0079] In some embodiments, the device provided by the embodiments of the present application can be realized in software, Figure 2 A face image processing device 255 stored in the memory 250 is shown, which can be software in the form of programs and plug-ins, including the following software modules: face key point detection model 2551, determination module 2552, interpolation fitting module 2553, and processing module 2554. These modules are logical, and thus can be combined or further split according to the functions implemented.

[0080] The functions of the various modules will be described below.

[0081] In some embodiments, the device provided by the embodiments of the present application can be implemented in a hardware manner. For example, the device provided by the embodiments of the present application can be a hardware decoding processor programmed to execute the face image processing method provided by the embodiments of the present application. For example, the hardware decoding processor can be implemented by one or more application specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field programmable gate arrays (FPGAs), or other electronic elements.

[0082] The face image processing method provided by the embodiments of the present application will be described in combination with an exemplary application and implementation of a server provided by the embodiments of the present application.

[0083] Referring to Figure 3 , Figure 3 is an optional flowchart of the face image processing method provided by the embodiments of the present application, which will be described in combination with the steps shown in Figure 3 .

[0084] In S101, the face image processing device acquires a face image to be processed, and performs face key point detection on the face image to be processed to obtain a face key point set.

[0085] The face image processing method provided by the embodiments of the present application can be applied to a scenario of using artificial intelligence technology to process an image containing a face, for example, face beautification, online makeup and beautification, face video synthesis, face recognition, and the like.

[0086] In S101, the face image processing device acquires a face image to be processed, and performs face key point detection on the face image to be processed to obtain a face key point set.

[0087] In S101, the face key point detection is to detect and locate the key parts of the face, including eyebrows, eyes, nose, mouth, face contour, etc., from the given face image, i.e., the face image to be processed, and mark the key points on the positions of the detected key parts. For example, a plurality of feature points are marked on the contour lines of the detected parts such as eyes and nose, and each feature point contains coordinate position information and semantic information in the face image to be processed. Here, the semantic information represents the key part where the feature point is located.

[0088] In some embodiments, the face image processing apparatus can detect a set of face key points from the face image to be processed by using a face key point detection model or a face feature point model. Here, the face key point detection model or the face feature point model is a neural network model trained based on labeled face image sample data by deep machine learning. For example, it can be a 68-point face feature point model based on the Dlib library for face detection, or a 96-point or 106-point face feature point model, etc. The specific selection is made according to the actual situation, and the embodiments of the present application are not limited. The embodiments of the present application show the 106 face key points output by the 106-point face feature point model on the face image to be processed in Figure 4

[0089] Here, by Figure 4 It can be seen that the set of face key points detected by the face key point detection does not contain the feature points of the forehead region, and the feature points of the forehead region need to be calculated by the method in the embodiments of the present application to obtain the key points of each part of the face and improve the accuracy of face image processing.

[0090] In S102, based on the first feature point representing the nasal root and the second feature point representing the lower jaw in the set of face key points, the first forehead feature point is determined. The first forehead feature point represents the highest point of the forehead on the external contour of the face.

[0091] In S102, the face image processing apparatus can determine the first feature point representing the nasal root and the second feature point representing the lower jaw according to the semantic information of each face key point in the set of face key points, and determine the highest point of the forehead region on the external contour of the face in the face image to be processed, i.e., the first forehead feature point, according to the positions of the first feature point and the second feature point and the distance therebetween, in combination with the structure and face proportion of a normal face.

[0092] In some embodiments, when the set of face key points contains Figure 4 ​The illustrated 106 facial key points, wherein 0-32 points mark the outer contour of the face, 33-37 points mark the left eyebrow upper contour, 38-42 points mark the right eyebrow upper contour, 43-51 points mark the nasal midline and lower nasal contour, 52-57 points mark the left eye contour, 58-63 points mark the right eye contour, and so on. Each facial key point contains its own semantic information, such as the semantic information of point 49 is the nasal root, the semantic information of point 0 is the leftmost outer face, the semantic information of point 32 is the rightmost outer face, the semantic information of point 16 is the chin or lower jaw, and so on. The face image processing apparatus can determine point 49 as the first feature point and point 16 as the second feature point according to the semantic information of each facial key point.

[0093] Here, the semantic information of the facial key points detected by different facial key point detection methods or network models may be slightly different, and in actual application, it is not limited to determining the first feature point or the second feature point according to the literal meaning of the semantic information of the facial key points. The first feature point or the second feature point can also be determined according to the actual semantic information of the facial key points or other information contained in the facial key points that can represent the point as the nasal root or the chin position. The first feature point or the second feature point is selected according to the actual situation, and the embodiments of the present application are not limited.

[0094] In some embodiments, based on the prior knowledge of the normal face structure, the highest point of the forehead can be located on the straight line passing through the nasal root point and the chin point. In the case where the first feature point representing the nasal root and the second feature point representing the chin are determined, the face image processing apparatus can determine the first forehead feature point, i.e. the position of the highest point of the forehead, on the vector pointing from the second feature point to the first feature point, based on the distance between the first feature point and the second feature point and the normal face ratio, for example, the distance ratio between the distance from the highest point of the forehead to the nasal root and the distance from the nasal root to the chin.

[0095] In some embodiments, the face image processing apparatus can calculate the horizontal coordinate and the vertical coordinate of the first forehead feature point in the face image to be processed through formula (1) and formula (2), thereby determining the first forehead feature point, as follows:

[0096] (1)

[0097] (2)

[0098] In formula (1) and formula (2), is a preset adjustment factor, the coordinates of the first feature point are , and the coordinates of the second feature point are The face image processing apparatus can multiply the preset adjustment factor with the horizontal coordinate and the vertical coordinate of the first feature point respectively to obtain a first horizontal product and a first vertical product ; and calculate a difference between the preset threshold and the preset adjustment factor, and multiply the difference with the horizontal coordinate and the vertical coordinate of the second feature point respectively to obtain a second horizontal product and a second vertical product ; the formula (1) and the formula (2) show the case that the preset threshold is 1; the face image processing apparatus takes the sum of the first horizontal product and the second horizontal product as the horizontal coordinate of the first forehead feature point, and takes the sum of the first vertical product and the second vertical product as the vertical coordinate of the first forehead feature point, where the preset adjustment factor is a value greater than 1, which is used to constrain the coordinates of the first forehead feature point on the vector from the second feature point to the first feature point.

[0099] In some embodiments, the face image processing apparatus can calculate the first forehead feature point according to the 43 points representing the root of the nose and the 16 points representing the lower jaw in the face key point set Figure 4 , and mark the first forehead feature point as the 109 point, as shown in Figure 5 .

[0100] In some embodiments, the face image processing apparatus can also determine the position of the highest forehead point as the first forehead feature point according to the distance, proportion or direction between the selected face key points based on the first feature point and the second feature point and the face key points representing other key parts in the face key point set, such as the face key points representing the eyebrows, eyes and the like, according to the similar geometric calculation method, and the specific selection is based on the actual situation, which is not limited in the embodiments of the present application.

[0101] In some embodiments, the face image processing apparatus can also determine the first forehead feature point in the preset range interval near the vector from the second feature point to the first feature point, and the specific selection is based on the actual situation, which is not limited in the embodiments of the present application.

[0102] S103, based on the first forehead feature point and the first feature point, respectively combining the third feature point representing the left side of the face contour and the fourth feature point representing the right side of the face contour, to determine the second forehead feature point and the third forehead feature point; the third feature point and the fourth feature point belong to the face key point set.

[0103] In S103, the face image processing apparatus can determine the third feature point representing the left side of the face contour, such as the 0th point in Figure 4 , and the fourth feature point representing the right side of the face contour, such as the 1st point in Figure 4Point 32. The face image processing device can then determine a second forehead feature point located between the outer side of the left face contour and the highest point of the forehead, based on the first forehead feature point and the first feature point, combined with the third feature point; and, based on the first forehead feature point and the first feature point, combined with the fourth feature point representing the outer side of the right face contour, determine a third forehead feature point located between the outer side of the right face contour and the highest point of the forehead.

[0104] In some embodiments, the face image processing device can calculate a first feature vector and a first length pointing from the first feature point to the first forehead feature point, and a second feature vector and a second length pointing from the first feature point to the third feature point, based on the coordinates of the first feature point and the first forehead feature point. The face image processing device can calculate the included angle between the first and second feature vectors, and determine a first intermediate feature vector based on the included angle; the face image processing device can also calculate a first average value of the first and second lengths. Thus, the face image processing device can determine the first intermediate feature vector based on the first intermediate feature vector. Compared with the first average The coordinates of the second forehead feature point are obtained through calculation, thus determining the second forehead feature point.

[0105] For example, such as Figure 6 As shown, when the first feature point is 43, the first forehead feature point is 109, and the third feature point is 0, the face image processing device can calculate the first feature vector pointing from the 43 point to the 109 point. , and the second eigenvector pointing from point 43 to point 0 The first intermediate feature vector is determined based on the included angle between the two points. ,in, The face image processing device calculates the distance between points 43 and 109 as the first length. Calculate the distance between point 43 and point 0 as the second length. ,calculate and First average ,Right now Furthermore, the facial image processing device can, according to... and The second forehead feature point was identified and marked as point 111.

[0106] It should be noted that the method for calculating the intermediate included angle and the average length in the embodiments of the present application is an example of calculating a first intermediate feature vector and a first average value according to the first feature point, the second feature point, and the first forehead feature point, and then obtaining the second forehead feature point. In actual use, the calculation method can be adjusted according to the actual situation, such as adjusting the angle calculation or length calculation method according to the face ratio of the character image, and the like. The specific selection is based on the actual situation, and the embodiments of the present application are not limited.

[0107] Similarly, the face image processing apparatus can calculate a third feature vector of the first feature point pointing to the fourth feature point and a third length; and calculate an intermediate included angle of the first feature vector and the third feature vector, so as to determine a second intermediate feature vector; calculate a second average value of the first length and the third length; and determine a third forehead feature point according to the second intermediate feature vector and the second average value. For example, as shown in FIG. 6, the fourth feature point is 32, the face image processing apparatus can calculate a first feature vector of 43 pointing to 109, a third feature vector of 43 pointing to 32 Figure 7 , and an intermediate included angle between the first feature vector and the third feature vector , so as to determine a second intermediate feature vector , and calculate a first length between 43 and 109 , a third length between 43 and 32 , calculate a second average value of and ; and then, according to and , determine a third forehead feature point, marked as 107. It should be noted that the embodiments of the present application do not limit the execution order of the face image processing apparatus for calculating the second forehead feature point and the third forehead feature point, which can be any sequence or parallel execution. The specific selection is based on the actual situation.

[0108] In some embodiments, the face image processing apparatus can also determine the second forehead feature point and the third forehead feature point according to the first feature point, the first forehead feature point, and other face key points, for example, 36 on the upper contour of the left eyebrow, 39 on the upper contour of the right eyebrow, and the like, in combination with the third feature point and the fourth feature point, respectively, to determine the second forehead feature point and the third forehead feature point according to the unit feature vector and the length between the feature points. The specific calculation can be performed according to the selected face key points in the actual situation, and the embodiments of the present application are not limited.

[0109] Figure 4

[0110] ​​S104, based on the first forehead feature point, the second forehead feature point and the third forehead feature point, interpolation fitting is performed to obtain a forehead feature point set; the forehead feature point set represents a forehead contour corresponding to the face to be processed.

[0111] In S104, after the first forehead feature point, the second forehead feature point and the third forehead feature point are determined, the face image processing apparatus can take the first forehead feature point, the second forehead feature point and the third forehead feature point as anchor points for preliminary positioning of the forehead region in the face image to be processed, and perform interpolation calculation in combination with adjacent face key points such as the third feature point and the fourth feature point to obtain more forehead feature points, and then obtain the forehead feature point set.

[0112] In the embodiments of the present application, interpolation calculation refers to fitting new feature points between specified feature points, so that the curve of the forehead region is smoother, and the connection with other regions such as the face contour is smoother. The face image processing apparatus can obtain a forehead curve constraint corresponding to the face to be processed according to the third feature point, the second forehead feature point, the first forehead feature point, the third forehead feature point and the fourth feature point; here, the forehead curve constraint can be a curve function representing the forehead curvature, and the face image processing apparatus can obtain a fitting relationship for fitting new feature points according to the forehead curve constraint, so as to perform interpolation fitting between the third feature point, the second forehead feature point, the first forehead feature point, the third forehead feature point and the fourth feature point based on the forehead curve constraint, and obtain the forehead feature point set containing the first forehead feature point, the second forehead feature point and the third forehead feature point.

[0113] In some embodiments, interpolation calculation can use Catmull-Rom polynomial fitting method, or other interpolation methods such as bilinear, cube, etc. through specified points, which are selected according to actual conditions, and the embodiments of the present application are not limited. In addition, the number of interpolation between specified feature points is not limited in the embodiments of the present application, and any integer greater than 1 can be specified.

[0114] In some embodiments, based on Figure 7 , the face image processing apparatus can perform interpolation calculation between the 0 point, the 111 point, the 109 point, the 107 point and the 32 point to obtain the forehead feature points of the 112 point, the 110 point, the 108 point and the 106 point shown in Figure 8 , and the face image processing apparatus takes the 106 point-112 point as the forehead feature point set. It can be seen that the 106 point-112 point marks the contour of the forehead region of the face to be processed.

[0115] S105, based on the forehead feature point set, performing image processing on the face image to be processed to obtain an image processing result.

[0116] In S105, based on the set of forehead feature points obtained in the foregoing steps, the face image processing apparatus can obtain feature points of all key parts in the forehead region of the face image to be processed, so that a higher-accuracy image processing process can be further performed to obtain an image processing result.

[0117] In some embodiments, the face image processing apparatus can perform image processing on the forehead region in the face to be processed based on the set of forehead feature points, such as sticker, beauty, virtual makeup effect generation, and the like. The face image processing apparatus can also perform image processing on the entire face region in combination with the set of forehead feature points and the set of face key points, such as at least one of face segmentation, face alignment, face recognition, face synthesis, and the like, to obtain an image processing result.

[0118] For example, the face image processing apparatus can perform face segmentation on the entire image containing a background image based on the set of forehead feature points and the set of face key points to obtain the entire face region, implement face cutout and the like, and further perform more image processing operations based on the segmented entire face region, such as superimposition of a virtual makeup effect to implement makeup and beauty functions, or replace the background image other than the entire face region, or replace the entire face region with another image to implement face replacement or face occlusion and the like. The specific selection is based on actual conditions, and the embodiments of the present application are not limited.

[0119] For example, the face image processing apparatus can generate a virtual image corresponding to the face image to be processed, such as a virtual character or a cartoon image similar to the real face, through face alignment based on the key parts labeled by the set of forehead feature points and the set of face key points, so that the virtual image presents similar facial features to the real face, and implements functions such as an animated character obtained by pinching the face of a real person in a game or a video, or a cartoon character expression driven by a real person's expression, and the like. The specific selection is based on actual conditions, and the embodiments of the present application are not limited.

[0120] For example, the face image processing apparatus can also perform image processing on at least one key part extracted from the face based on the set of forehead feature points and the set of face key points, or perform face synthesis on one or more facial features of a person 1 and one or more facial features of a person 2, to implement functions such as the effect of different facial parts after beauty in the beauty industry presented to the user, or face effect generation in a video editing application, and the like. Alternatively, the face image processing apparatus can also perform face recognition, expression recognition, and the like based on the set of forehead feature points and the set of face key points, and the specific selection is based on actual conditions, and the embodiments of the present application are not limited.

[0121] It can be understood that, in the embodiments of the present application, according to the key points in the face key point set which are not in the forehead region, such as the feature points representing the root of the nose, the left side of the face contour and the right side of the face contour, the first forehead feature point, the second forehead feature point and the third forehead feature point on the forehead contour can be preliminarily located; and on this basis, more forehead feature points are calculated by using the adaptive interpolation method, so as to obtain a forehead feature point set corresponding to a smoother forehead contour. The calculation process of the embodiments of the present application is faster, and for various different face shapes, robust forehead feature points can be determined by the adaptive interpolation method, thereby improving the efficiency and accuracy of forehead feature point recognition, and further improving the efficiency and accuracy of face image processing based on the forehead feature point set.

[0122] In some embodiments, the applicant finds that, compared with the face image to be processed collected at a normal angle, in the face image to be processed collected in the case of lifting the head, the distance between the first feature point representing the root of the nose, such as the 43 point, and the second feature point representing the chin, such as the 16 point, will become larger, and the distance between the first feature point and the first forehead feature point, such as the 109 point, will become smaller; in the face image to be processed collected in the case of lowering the head, the distance between the first feature point and the second feature point will become smaller, and the distance between the first feature point and the first forehead feature point will become larger. In order to make the forehead feature point set adapt to the changes of the face at various angles, the embodiments of the present application can dynamically set a preset adjustment factor to calculate the first forehead feature point, and further obtain the forehead feature point set. The process of dynamically setting the preset adjustment factor can be realized by executing S001-S002, as follows:

[0123] S001, calculate a first distance between the second feature point and the first feature point, and calculate a second distance between a preset fixed distance point and the first feature point; the distance between the preset fixed distance point and the first feature point changes less than a preset change threshold in the case of face angle change; the preset fixed distance point belongs to the face key point set.

[0124] S002, obtain a preset adjustment factor based on the ratio of the first distance to the second distance.

[0125] In the embodiments of the present application, in the case of face angle change, the distance between the preset fixed distance point in the face key point set and the first feature point changes less than a preset change threshold. Exemplarily, through a large number of experiments, it is found that, in the case of face angle change, the distance between the 16 point and the 43 point changes less than 5 pixels, and the distance between the 16 point and the 109 point changes less than 5 pixels. Figure 4In the set of facial key points shown, the distance between the 49th point representing the tip of the nose and the 43rd point representing the root of the nose, i.e., the first feature point, does not change substantially with the rotation of the face. Therefore, the face image processing apparatus calculates the 49th point as a preset fixed distance point, and calculates a first distance between the second feature point and the first feature point, and a second distance between the preset fixed distance point and the first feature point. Then, based on the ratio of the first distance to the second distance, a preset adjustment factor is obtained.

[0126] In some embodiments, the above calculation process can be implemented by formula (3) as follows:

[0127] (3)

[0128] In formula (3), is the coordinate of the preset fixed distance point, is the first distance between the second feature point and the first feature point, is the second distance between the preset fixed distance point and the first feature point, and is a preset adjustment parameter, which is used to make the first forehead feature point calculated according to r remain on the vector pointing from the second feature point to the first feature point. Wherein, may be a value greater than 1; may be a value less than 1 and greater than 0; for example, may be 1.35, may be 0.8.

[0129] It should be noted that the specific feature points used to calculate the preset adjustment factor are not limited in the embodiments of the present application, and other two points whose distance does not change with the rotation of the face can also be introduced, and the values of r and d can be adjusted accordingly to calculate the preset adjustment factor. and

[0130] It can be understood that by introducing the preset fixed distance point that is irrelevant to the change of the face angle to dynamically set the preset adjustment factor, and then calculating the first forehead feature point according to the preset adjustment factor, the first forehead feature point with high accuracy can be calculated in the case of lifting the head, lowering the head, etc. of the face, and various face angle changes are adapted, thereby improving the accuracy of face image processing.

[0131] In some embodiments, referring to Figure 9 , Figure 9 is an optional flowchart of the method provided by the embodiments of the present application, Figure 3 S105 in the above formula can be implemented by executing S1051-S1054, which will be described in conjunction with each step.

[0132] ​S1051, divide the to-be-processed face image into a plurality of real face regions according to the face key point set and the forehead feature point set.

[0133] In S1051, the face image processing apparatus can perform grid division on the to-be-processed face according to the feature points contained in the face key point set and the forehead feature point set, divide the to-be-processed face image into a plurality of real face regions. Here, it can be seen that through grid division, the face image processing apparatus divides the to-be-processed face image into more detailed texture regions, and compared with the method of performing affine transformation according to the semantic segmentation of each face key part, the plurality of real face regions obtained in the division stage in the embodiment of the present application have finer granularity, which helps to improve the accuracy of face image processing based on the plurality of real face regions.

[0134] In some embodiments, the face image processing apparatus can take each feature point in the face key point set and the forehead feature point set as a vertex, perform triangular mesh division by using a mesh division algorithm, and obtain a plurality of real face regions composed of triangular meshes.

[0135] In some embodiments, the face image processing apparatus can also perform other forms of mesh division, such as quadrilateral mesh division, to divide the to-be-processed face image. The specific selection is made according to actual conditions, and the embodiment of the present application is not limited.

[0136] In some embodiments, the mesh division algorithm can be a Delaunay triangulation algorithm, or a Loop algorithm, a Doo-Sabine algorithm, or a Catmull-Clark algorithm, etc. The specific selection is made according to actual conditions, and the embodiment of the present application is not limited.

[0137] S1052, obtain a preset special effect face corresponding to the to-be-processed face image; the preset special effect face is a face template containing a preset special effect image; the preset special effect face contains a plurality of preset face regions obtained by performing the same feature point calculation and division processing on the face template.

[0138] In S1052, the face image processing apparatus can obtain a preset special effect face for performing special effect processing on the to-be-processed face image. Here, the preset special effect face is a face template containing a preset special effect image; for example, the face template can be a template generated according to a standard face, the preset special effect image can be a virtual makeup, and the preset special effect face can be a virtual makeup applied to the template of the standard face.

[0139] In the embodiments of the present application, the face image processing apparatus can also perform face key point detection on the preset special effect face through the same process in S101-S104, obtain a special effect face key point set corresponding to the preset special effect face, and calculate a special effect forehead feature point set corresponding to the preset special effect face based on the special effect face key point set. Here, when performing face key point detection and interpolation calculation, the face image processing apparatus can process the preset special effect face according to the same number of feature points as the face image to be processed, so that the total number of feature points in the special effect face key point set and the special effect forehead feature point set corresponding to the preset special effect face is consistent with the total number of feature points in the face image to be processed. Further, the face image processing apparatus can perform grid division on the preset special effect face according to the special effect face key point set and the special effect forehead feature point set through the same grid division method as in S1051, so that the preset special effect face obtained by division contains multiple preset face regions one-to-one corresponding to multiple real face regions of the face image to be processed. For example, the triangular region composed of points 33, 34 and 64 in the multiple real face regions, and the triangular region composed of points 33, 34 and 64 in the corresponding preset face region.

[0140] In some embodiments, the preset special effect image can also be other special effect images such as stickers, filters, etc. The specific selection is made according to actual conditions, and the embodiments of the present application are not limited.

[0141] In some embodiments, the face image processing apparatus can perform feature point calculation and division processing on the preset feature face once, and save the obtained feature point label distribution and divided grid region. Subsequently, the saved preset feature face data can be directly obtained and fused with different face images to be processed.

[0142] S1053, according to the correspondence between the multiple real face regions and the multiple preset face regions, obtaining a special effect pixel corresponding to each face pixel in the face image to be processed in the preset special effect face.

[0143] In S1053, since the multiple real face regions and the multiple preset face regions are one-to-one corresponding, for each face pixel in the face image to be processed, the face image processing apparatus can sample each face pixel in the corresponding preset face region according to the correspondence between the multiple real face regions and the multiple preset face regions, to obtain a special effect pixel corresponding to each face pixel in the preset special effect face.

[0144] In some embodiments, each face pixel can determine its relative position in the real face region where it is located according to the vertex coordinates of the real face region. The face image processing apparatus can determine the relative position of each face pixel in the target real face region according to the vertex positions of the target real face region where each face pixel is located; and take the real face region where each face pixel is located as the target real face region, and take the preset face region corresponding to the target real face region as the target preset face region according to the correspondence between the plurality of real face regions and the plurality of preset face regions, that is, determine the target preset face region corresponding to the target real face region; and the face image processing apparatus can determine the pixel in the target preset face region corresponding to the relative position of the target real face region as the special effect pixel corresponding to each face pixel according to the vertex positions in the target preset face region. In this way, the face image processing apparatus obtains the special effect pixel corresponding to each face pixel in the preset special effect face.

[0145] Here, it should be noted that, since the pixel resolution of the face image to be processed and the pixel resolution of the preset special effect face can be different, the correspondence between the face pixel and the special effect pixel is not limited to a one-to-one correspondence. For example, when the pixel resolution of the face image to be processed is higher than the pixel resolution of the preset special effect face, there can be a case where at least one face pixel corresponds to the same special effect pixel in the preset special effect face. The embodiments of the present application are not limited.

[0146] In S1054, the face image processing apparatus can perform pixel fusion on each face pixel and the special effect pixel corresponding thereto, superimpose the corresponding special effect pixel on each face pixel, and further obtain an image processing result in which the face image to be processed and the preset special effect image are superimposed.

[0147] In S1054, the face image processing apparatus can perform pixel fusion on each face pixel and the special effect pixel corresponding thereto, superimpose the corresponding special effect pixel on each face pixel, and further obtain an image processing result in which the face image to be processed and the preset special effect image are superimposed.

[0148] In some embodiments, the face image processing apparatus can perform pixel fusion by means of positive-to-negative superimposition, normal fusion, strong light fusion, etc. The specific fusion manner is not limited in the embodiments of the present application. The face image processing apparatus can perform chroma fusion on the color information, such as RGB value, of each face pixel and the special effect pixel corresponding thereto, or perform pixel fusion on the grayscale information, luminance information, etc. of each face pixel and the special effect pixel corresponding thereto. The specific selection is made according to the actual situation, and the embodiments of the present application are not limited.

[0149] It can be understood that, by means of the grid construction, the embodiments of the present application can obtain more detailed topological structure of the to-be-processed face and the texture of the special effect image, such as the topological structure of the makeup texture, and further obtain the texture mapping relationship of the two. Compared with the current affine mapping method, the mapping relationship obtained by the subdivision grid is more accurate, so that the makeup texture is more fitted to the face in various postures, the makeup is more natural, and the accuracy of the face image processing is improved.

[0150] In some embodiments, for S1053 and S1054 described above, the face image processing apparatus can determine the corresponding special effect pixel of each face pixel point in parallel through a graphics processing module, such as a graphics processing unit (GPU), and perform parallel fusion processing on each face pixel point and its corresponding special effect pixel to obtain the image processing result.

[0151] In some embodiments, the face image processing apparatus can perform chroma fusion on each face pixel point and its corresponding special effect pixel to obtain an intermediate chroma, as shown in formula (4), as follows:

[0152] (4)

[0153] In formula (4), is the RGB value of each face pixel, is the RGB value of the special effect pixel corresponding to each face pixel, is the intermediate chroma.

[0154] The face image processing apparatus adjusts the fusion intensity of the chroma of each face pixel point and the intermediate chroma by a preset fusion intensity factor to obtain a first adjustment result and a second adjustment result; and obtains the image fusion result corresponding to each face pixel point by combining the first adjustment result and the second adjustment result as the image processing result, as shown in formula (5), as follows:

[0155] (5)

[0156] In formula (5), is the preset fusion intensity factor, is the first adjustment result, is the second adjustment result, is the image fusion result corresponding to each face pixel point. The face image processing apparatus can obtain the image fusion result corresponding to each face pixel point simultaneously as the image processing result by parallel processing of the same process on each face pixel point.

[0157] It can be understood that, by means of the graphic processing module, the application embodiment can process sampling and fusion between the makeup texture and the to-be-processed face in parallel, so that the calculation amount and processing time of the central processing unit on the electronic device can be greatly reduced, thereby improving the efficiency of face image processing and meeting the real-time requirements of users in video and live scenes.

[0158] In the following, an exemplary application of the application embodiment in an actual application scenario will be described.

[0159] The application embodiment can be applied in online makeup function in video live streaming. A user such as a video host can record live streaming content in real time through a live streaming application on a mobile phone or other live streaming terminal, and upload the live streaming content to a background server of the live streaming application through a network, and the background server sends the live streaming content to a viewer terminal used by a viewer for presentation of the live streaming content. When applied to implement the online makeup function, the face image processing device provided by the application embodiment can be deployed in the background server of the live streaming application, and includes an identification module, an adaptive interpolation module, a mapping module and a fusion module as shown in the figure. Figure 10 The face image processing device can use the identification module, the adaptive interpolation module, the mapping module and the fusion module to implement the application of online makeup by executing the processes of S201-S204 as shown in the figure, as follows: Figure 11

[0160] S201, detecting key points of a to-be-processed face in a current live streaming video frame by means of the identification module to obtain a face key point set.

[0161] In S201, when it is detected that the video host has enabled the online makeup function in the live streaming application, the face image processing device can use the 106-point face key point detection model contained in the identification module to perform face key point detection on the to-be-processed face image in the current live streaming video frame in the video stream uploaded by the live streaming terminal, and obtain 106 face key points labeled with different semantics, i.e., a face key point set. Here, the 106 face key points do not include feature points in the forehead region.

[0162] S202, calculating a forehead feature point set according to the face key point set by means of the adaptive interpolation module.

[0163] Here, the process of S202 is consistent with the process description in S102-S104 described above, and will not be described again here.

[0164] S203, obtaining a mapping relationship between each face pixel in the to-be-processed face image and a virtual makeup pixel by means of the mapping module.

[0165] Here, the virtual makeup pixel is equivalent to a special effect pixel, and the execution process of S203 is consistent with the description of S1051-S1053, and will not be described again here.​

[0166] S204, by the fusion module, each face pixel is fused with virtual makeup pixel, obtains the image processing result of applying virtual makeup on the to-be-processed face image.

[0167] Here, the execution process of S204 is consistent with that described in S1054, which will not be repeated here.

[0168] It can be understood that the face image processing method provided by the embodiments of the present application can be applied to the scene of face beautification by virtual makeup. The feature points of the forehead region can be fitted by adaptive interpolation of the face key point model to obtain a more robust set of forehead feature points. Moreover, through the grid construction method, the mapping relationship between the to-be-processed face and the makeup texture can be more accurately obtained. Moreover, through the calculation of the graphics rendering method, a more natural and natural fusion effect can be obtained in the fusion module, so as to realize real-time and accurate virtual makeup effect, reduce the user's makeup threshold, and save the time and cost of makeup materials, provide real-time and accurate virtual makeup for users, and make the virtual makeup more natural and comfortable, meet the user's makeup needs, and improve the user's satisfaction and stickiness to the platform.

[0169] The following continues to illustrate an example structure of the face image processing apparatus 255 provided by the embodiments of the present application as a software module. In some embodiments, as shown in Figure 2 The software module stored in the face image processing apparatus 255 of the storage 250 can include:

[0170] The face key point detection model 2551 is configured to perform face key point detection on the to-be-processed face image to obtain a set of face key points.

[0171] The determination module 2552 is configured to determine a first forehead feature point based on a first feature point representing the nasal root and a second feature point representing the lower jaw in the set of face key points; the first forehead feature point represents the highest point of the forehead on the external contour of the face; and determine a second forehead feature point and a third forehead feature point based on the first forehead feature point and the first feature point, respectively, in combination with a third feature point representing the left face contour outside and a fourth feature point representing the right face contour outside; the third feature point and the fourth feature point belong to the set of face key points.

[0172] The interpolation fitting module 2553 is configured to perform interpolation fitting based on the first forehead feature point, the second forehead feature point and the third forehead feature point to obtain a set of forehead feature points; the set of forehead feature points represents the forehead contour corresponding to the to-be-processed face.

[0173] The processing module 2554 is configured to perform image processing on the face image to be processed based on the set of forehead feature points, and obtain an image processing result.

[0174] In some embodiments, the determining module 2552 is further configured to determine the first forehead feature point on a vector pointing from the second feature point to the first feature point based on a distance between the first feature point and the second feature point.

[0175] In some embodiments, the determining module 2552 is further configured to multiply a preset adjustment factor by the horizontal coordinate and the vertical coordinate of the first feature point respectively to obtain a first horizontal product and a first vertical product; the preset adjustment factor is a value greater than 1; calculate a difference between a preset threshold and the preset adjustment factor, and multiply the difference by the horizontal coordinate and the vertical coordinate of the second feature point respectively to obtain a second horizontal product and a second vertical product; and take a sum of the first horizontal product and the second horizontal product as the horizontal coordinate of the first forehead feature point, and take a sum of the first vertical product and the second vertical product as the vertical coordinate of the first forehead feature point, thereby determining the first forehead feature point.

[0176] In some embodiments, the determining module 2552 is further configured to calculate a first distance between the second feature point and the first feature point, and calculate a second distance between a preset fixed distance point and the first feature point; the distance between the preset fixed distance point and the first feature point changes less than a preset change threshold in the case of a change in the angle of the face; the preset fixed distance point belongs to the set of face key points; and obtain the preset adjustment factor based on a ratio of the first distance to the second distance.

[0177] In some embodiments, the determining module 2552 is further configured to calculate a first feature vector and a first length of the first feature point pointing to the first forehead feature point, a second feature vector and a second length of the first feature point pointing to the third feature point, and a third feature vector and a third length of the first feature point pointing to the fourth feature point; calculate a middle included angle between the first feature vector and the second feature vector to determine a first intermediate feature vector; calculate a first average value of the first length and the second length; determine the second forehead feature point according to the first intermediate feature vector and the first average value; calculate a middle included angle between the first feature vector and the third feature vector to determine a second intermediate feature vector; calculate a second average value of the first length and the third length; and determine the third forehead feature point according to the second intermediate feature vector and the second average value.

[0178] In some embodiments, the interpolation fitting module 2553 is further configured to obtain a forehead curve constraint corresponding to the face to be processed according to the third feature point, the second forehead feature point, the first forehead feature point, the third forehead feature point, and the fourth feature point; and perform interpolation fitting between the third feature point, the second forehead feature point, the first forehead feature point, the third forehead feature point, and the fourth feature point based on the forehead curve constraint to obtain the forehead feature point set containing the first forehead feature point, the second forehead feature point, and the third forehead feature point.

[0179] In some embodiments, the processing module 2554 is further configured to divide the face to be processed into a plurality of real face regions according to the face key point set and the forehead feature point set; obtain a preset special effect face corresponding to the face to be processed; the preset special effect face is a face template containing a preset special effect image; the preset special effect face contains a plurality of preset face regions obtained by performing the same feature point calculation and division processing on the face template in advance; obtain a special effect pixel corresponding to each face pixel in the face to be processed in the preset special effect face according to the correspondence between the plurality of real face regions and the plurality of preset face regions; and perform pixel fusion on the each face pixel and the corresponding special effect pixel to obtain an image processing result in which the face to be processed and the preset special effect image are superimposed.

[0180] In some embodiments, the processing module 2554 is further configured to perform triangular mesh division on the face key point set and the forehead feature point set by using a mesh division algorithm to obtain the plurality of real face regions formed by triangular meshes.

[0181] In some embodiments, the processing module 2554 is further configured to, for each face pixel, perform weighted calculation according to the vertex position of a target real face region in which the each face pixel is located to obtain a relative position of the each face pixel in the target real face region; determine a target preset face region corresponding to the target real face region according to the correspondence between the plurality of real face regions and the plurality of preset face regions; and determine a pixel corresponding to the relative position in the target preset face region as a special effect pixel corresponding to the each face pixel according to the vertex position of the target preset face region.

[0182] In some embodiments, the processing module 2554 is further configured to perform parallel fusion processing on the each face pixel and the corresponding material pixel by using a graphics processing module to obtain the image processing result.

[0183] In some embodiments, the processing module 2554 is further configured to perform chroma fusion on each face pixel point and the corresponding material pixel point to obtain intermediate chroma; perform fusion intensity adjustment on the chroma of each face pixel point and the intermediate chroma by using a preset fusion intensity factor to obtain a first adjustment result and a second adjustment result; and combine the first adjustment result and the second adjustment result to obtain the image fusion result corresponding to each face pixel point as the image processing result.

[0184] In some embodiments, the processing module 2554 is further configured to perform at least one of face segmentation, face alignment, face recognition, and face synthesis on the face image to be processed based on the set of forehead feature points to obtain the image processing result.

[0185] It should be noted that the above description of the device embodiments is similar to the description of the above method embodiments, and has similar beneficial effects to the method embodiments. For technical details not disclosed in the device embodiments of the present application, please refer to the description of the method embodiments of the present application for understanding.

[0186] The embodiments of the present application provide a computer readable storage medium storing executable instructions, wherein the executable instructions, when executed by a processor, will cause the processor to perform the method provided by the embodiments of the present application, for example, the method shown in Figure 3 、 9

[0187] In some embodiments, the computer readable storage medium can be FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disc, or CD-ROM memory, etc. It can also be various devices including one or any combination of the above storage devices.

[0188] In some embodiments, the executable instructions can be in the form of programs, software, software modules, scripts or codes, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and can be deployed in any form, including being deployed as independent programs or being deployed as modules, components, subroutines or other units suitable for use in a computing environment.

[0189] ​By way of example, the executable instructions can correspond to a file in a file system, can be stored in a portion of a file that is used for other programs or data, can be stored as one or more separate files in a file system, or can be stored and implemented in a single file with other programs or data. Examples of files that might employ the executable instructions include, but are not limited to, programs, files, scripts, components, plug-ins, wed pages, icons, and various combinations thereof. By way of further example, executable instructions can be deployed to be executed on one computing device or on multiple computing devices that are located at one site or distributed across multiple sites and interconnected by a communication network.

[0190] In summary, the embodiment of the present application can preliminarily locate the first forehead feature point, the second forehead feature point and the third forehead feature point on the forehead contour according to the key points in the face key point set, such as the key points representing the nasal root, the left face contour outside and the right face contour outside; and on this basis, more forehead feature points are calculated by using the adaptive interpolation method, so as to obtain a forehead feature point set corresponding to a smoother forehead contour. The calculation process of the embodiment of the present application is faster, and robust forehead feature points can be determined by the adaptive interpolation method for various different face shapes, thereby improving the efficiency and accuracy of forehead feature point recognition, and further improving the efficiency and accuracy of face image processing based on the forehead feature point set. Moreover, the preset adjustment factor is dynamically set by introducing the preset fixed distance point irrelevant to the face angle change, and then the first forehead feature point is calculated according to the preset adjustment factor, so that the first forehead feature point with high accuracy can be calculated in the case of lifting the head, lowering the head and other situations of the face, various face angle changes are adapted, and the accuracy of face image processing is improved. Furthermore, by the grid construction method, a more detailed topology of the face to be processed and the texture of the special effect image, such as the makeup texture topology, can be obtained, and then the texture mapping relationship between the two is obtained. Compared with the current affine mapping method, the mapping relationship obtained by the subdivision grid in the embodiment of the present application is more accurate, so that the makeup texture is more fitted to the face in various postures, the makeup is more natural, and the accuracy of face image processing is improved. Moreover, by the graphic processing module, the sampling and fusion between the makeup texture and the face to be processed are processed in parallel, so that the calculation amount and processing time of the central processing unit on the electronic device can be greatly reduced, thereby improving the efficiency of face image processing and meeting the real-time requirements of users in video and live scenes.

[0191] The above merely describes the embodiments of the present application, but does not serve to limit the protection scope of the present application. Any modification, equivalent replacement and improvement made within the spirit and scope of the present application shall fall within the protection scope of the present application.

Claims

1. A face image processing method, characterized by, The method comprises the following steps: performing face key point detection on a face image to be processed to obtain a face key point set; calculating a first distance between a second feature point representing a lower jaw in the face key point set and a first feature point representing a root of a nose in the face key point set, and calculating a second distance between a tip of the nose and the first feature point; wherein the distance between the tip of the nose and the first feature point changes less than a preset change threshold in the case of a change in the angle of the face; determining a first forehead feature point on a vector pointing from the second feature point to the first feature point based on a preset adjustment factor and the distance between the first feature point and the second feature point; the preset adjustment factor is a ratio of the first distance to the second distance; the first forehead feature point represents the highest point on the forehead contour of the face; determining a second forehead feature point and a third forehead feature point based on the first forehead feature point and the first feature point, respectively in combination with a third feature point representing the outside of the left face contour and a fourth feature point representing the outside of the right face contour; the third feature point and the fourth feature point belong to the face key point set; performing interpolation fitting based on the first forehead feature point, the second forehead feature point, and the third forehead feature point to obtain a forehead feature point set; the forehead feature point set represents the forehead contour corresponding to the face image to be processed; performing image processing on the face image to be processed based on the forehead feature point set to obtain an image processing result.

2. The method of claim 1, wherein, The method comprises the following steps: multiplying the preset adjustment factor by the horizontal coordinate and the vertical coordinate of the first feature point respectively to obtain a first horizontal product and a first vertical product; the preset adjustment factor is a value greater than 1; calculating a difference between a preset threshold and the preset adjustment factor, and multiplying the difference by the horizontal coordinate and the vertical coordinate of the second feature point respectively to obtain a second horizontal product and a second vertical product; taking the sum of the first horizontal product and the second horizontal product as the horizontal coordinate of the first forehead feature point; taking the sum of the first vertical product and the second vertical product as the vertical coordinate of the first forehead feature point, thereby determining the first forehead feature point.

3. The method of claim 2, wherein, The tip of the nose belongs to the face key point set.

4. The method of claim 1, wherein, The method comprises the following steps: calculating a first feature vector and a first length of the first feature point pointing to the first forehead feature point, a second feature vector and a second length of the first feature point pointing to the third feature point, and a third feature vector and a third length of the first feature point pointing to the fourth feature point; calculating the intermediate angle between the first feature vector and the second feature vector to determine a first intermediate feature vector; calculating a first average value of the first length and the second length; determining the second forehead feature point according to the first intermediate feature vector and the first average value; calculating an intermediate included angle between the first feature vector and the third feature vector, thereby determining a second intermediate feature vector; calculating a second average value of the first length and the third length; determining the third forehead feature point according to the second intermediate feature vector and the second average value.

5. The method of claim 1, wherein, the interpolation fitting based on the first forehead feature point, the second forehead feature point and the third forehead feature point to obtain a forehead feature point set, comprising: obtaining a forehead curve constraint corresponding to the to-be-processed face according to the third feature point, the second forehead feature point, the first forehead feature point, the third forehead feature point and the fourth feature point; based on the forehead curve constraint, interpolation fitting is performed between the third feature point, the second forehead feature point, the first forehead feature point, the third forehead feature point and the fourth feature point to obtain the forehead feature point set containing the first forehead feature point, the second forehead feature point and the third forehead feature point.

6. The method according to any one of claims 1 to 5, characterized in that, the image processing result obtained by performing image processing on the to-be-processed face image based on the forehead feature point set, comprising: dividing the to-be-processed face image into a plurality of real face regions according to the face key point set and the forehead feature point set; obtaining a preset special effect face corresponding to the to-be-processed face image; the preset special effect face is a face template containing a preset special effect image; the preset special effect face contains a plurality of preset face regions obtained by performing the same feature point calculation and division processing on the face template; obtaining a special effect pixel corresponding to each face pixel in the to-be-processed face image in the preset special effect face according to the correspondence between the plurality of real face regions and the plurality of preset face regions; pixel fusion is performed on each face pixel and its corresponding special effect pixel to obtain an image processing result in which the to-be-processed face image and the preset special effect image are superimposed.

7. The method of claim 6, wherein, the to-be-processed face image is divided into a plurality of real face regions according to the face key point set and the forehead feature point set, comprising: each feature point in the face key point set and the forehead feature point set is taken as a vertex, and a triangular mesh division is performed by using a mesh division algorithm to obtain the plurality of real face regions formed by triangular meshes.

8. The method of claim 7, wherein, for each face pixel, a relative position of the each face pixel in the target real face region is obtained by weighted calculation according to the vertex position of the target real face region where the each face pixel is located; the target preset face region corresponding to the target real face region is determined according to the correspondence between the plurality of real face regions and the plurality of preset face regions; ​ According to the vertex position of the target preset face region, a pixel corresponding to the relative position in the target preset face region is determined as a special effect pixel corresponding to each face pixel.

9. The method of claim 6, wherein, The pixel fusion of each face pixel and the corresponding special effect pixel to obtain the image processing result of the superposition of the to-be-processed face image and the preset special effect image includes: The parallel fusion processing of each face pixel and the corresponding material pixel by the graphic processing module to obtain the image processing result.

10. The method of claim 9, wherein, The parallel fusion processing of each face pixel and the corresponding material pixel by the graphic processing module to obtain the image processing result includes: The chroma fusion of each face pixel and the corresponding material pixel to obtain intermediate chroma; The chroma of each face pixel and the intermediate chroma are respectively fused by a preset fusion intensity factor to obtain a first adjustment result and a second adjustment result; The image fusion result corresponding to each face pixel is obtained by combining the first adjustment result and the second adjustment result as the image processing result.

11. The method according to any one of claims 1 to 5, characterized in that, The image processing of the to-be-processed face image based on the forehead feature point set includes: At least one of face segmentation, face alignment, face recognition, and face synthesis is performed on the to-be-processed face image based on the forehead feature point set to obtain the image processing result.

12. A face image processing apparatus, characterized by comprising: It includes: A face key point detection model is used to detect face key points in a to-be-processed face image to obtain a face key point set. A determination module is configured to calculate a first distance between a second feature point representing a lower jaw in the face key point set and a first feature point representing a nasal root in the face key point set, and calculate a second distance between a tip of a nose in the face key point set and the first feature point; wherein the distance between the tip of the nose and the first feature point changes less than a preset change threshold in the case of face angle change. The determination module is configured to determine a first forehead feature point on a vector pointing from the second feature point to the first feature point based on a preset adjustment factor and the distance between the first feature point and the second feature point; the preset adjustment factor is the ratio of the first distance to the second distance; the first forehead feature point represents the highest point of the forehead on the external contour of the face; and based on the first forehead feature point and the first feature point, a second forehead feature point and a third forehead feature point are determined in combination with a third feature point representing the left face contour and a fourth feature point representing the right face contour; the third feature point and the fourth feature point belong to the face key point set. An interpolation fitting module is configured to perform interpolation fitting based on the first forehead feature point, the second forehead feature point, and the third forehead feature point to obtain a forehead feature point set; the forehead feature point set represents the forehead contour corresponding to the to-be-processed face. A processing module is configured to perform image processing on the to-be-processed face image based on the forehead feature point set to obtain an image processing result.

13. The apparatus of claim 12, wherein the determining module is further configured to multiply the preset adjustment factor by the horizontal coordinate and the vertical coordinate of the first feature point respectively to obtain a first horizontal product and a first vertical product, the preset adjustment factor being a value greater than 1; calculate a difference between the preset threshold and the preset adjustment factor, and multiply the difference by the horizontal coordinate and the vertical coordinate of the second feature point respectively to obtain a second horizontal product and a second vertical product; sum the first horizontal product and the second horizontal product as the horizontal coordinate of the first forehead feature point; sum the first vertical product and the second vertical product as the vertical coordinate of the first forehead feature point, thereby determining the first forehead feature point.

14. The apparatus of claim 12, wherein the determining module is further configured to calculate a first feature vector and a first length of the first feature point pointing to the first forehead feature point, a second feature vector and a second length of the first feature point pointing to the third feature point, and a third feature vector and a third length of the first feature point pointing to the fourth feature point; calculate an intermediate included angle between the first feature vector and the second feature vector, thereby determining a first intermediate feature vector; calculate a first average value of the first length and the second length; determine the second forehead feature point according to the first intermediate feature vector and the first average value; calculate an intermediate included angle between the first feature vector and the third feature vector, thereby determining a second intermediate feature vector; calculate a second average value of the first length and the third length; determine the third forehead feature point according to the second intermediate feature vector and the second average value. comprising: a memory configured to store executable instructions; 15. An electronic device, comprising: a processor configured to execute the executable instructions stored in the memory to implement the method of any one of claims 1 to 11. executable instructions stored in the memory, the executable instructions being configured to be executed by the processor to implement the method of any one of claims 1 to 11. ​ 16. A computer-readable storage medium, characterized in that, ​

Images