Method for processing a face image

Abstract

The scheme disclosed in the application belongs to the technical field of artificial intelligence, and particularly relates to a method for processing a face image. The method comprises: taking a first face image to be processed; extracting first face key points corresponding to the first face image; determining a plurality of software development kits (SDKs) for processing the first face image; processing the first face image based on a face key point conversion model and the SDKs to obtain a second face image processed by the plurality of SDKs, wherein the face key point conversion model is used to generate second face key points matched with each of the SDKs according to the first face key points. The application can improve the flexibility of processing a face image using an SDK.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method for processing human face images. Background Technology

[0002] Video processing applications often involve applying special effects to facial images included in the video. For example, in live streaming applications, it is common to apply beauty filters to the anchor's facial image or add decorations to the facial image.

[0003] Currently, video processing applications mainly achieve facial image effects processing through third-party SDKs (Software Development Kits) that process facial images.

[0004] Different SDKs produce different effects; for example, SDK1 provides a big eye effect, while SDK2 provides a sunglasses effect. However, different SDKs use different keypoint formats (including the number of keypoints and their relative positions within the image) when processing facial images. This results in poor compatibility between SDKs, meaning users can currently only select one SDK for each video processing application. Summary of the Invention

[0005] This application provides a method for processing facial images, capable of combining multiple SDKs to process facial images. The technical solution is as follows: In a first aspect, a method for processing a face image is provided. The method includes: acquiring a first face image to be processed; extracting first facial key points corresponding to the first face image; determining multiple software development kits (SDKs) for processing the first face image; and processing the first face image based on the first face image, a facial key point conversion model, and the multiple SDKs to obtain a second face image processed by the multiple SDKs, wherein the facial key point conversion model is used to generate second facial key points matching each SDK based on the first facial key points.

[0006] Optionally, the step of processing the first face image based on the first face image, the facial landmark conversion model, and multiple SDKs to obtain a second face image processed by the multiple SDKs includes: The first facial landmark is input into the facial landmark conversion model to obtain the second facial landmark output by the facial landmark conversion model that matches each of the multiple SDKs; Based on multiple SDKs and second facial key points matched with each SDK, the first face image is processed to obtain a second face image processed by multiple SDKs.

[0007] Optionally, multiple SDKs are used to process the overall image of the first face image. The process of processing the first face image based on the multiple SDKs and second facial key points matched with each SDK to obtain a second face image processed by the multiple SDKs includes: The first face image is processed according to the processing order of the multiple SDKs and the second facial key points matched by each SDK. The image processed by the first SDK among the multiple SDKs is the first face image, the images processed by the other SDKs among the multiple SDKs are the processing results of the corresponding previous SDKs, and the processing result of the last SDK among the multiple SDKs is the second face image.

[0008] Optionally, the plurality of SDKs are used to process local images of the first face image. The process of processing the first face image based on the plurality of SDKs and second facial key points matched with each SDK to obtain a second face image processed by the plurality of SDKs includes: Each SDK is controlled to process the first face image based on the matched second facial key points, so as to obtain a third face image after processing the first face image by each SDK; The multiple third face images are fused together to obtain the second face image.

[0009] Optionally, the multiple SDKs correspond to a processing order. The step of processing the first face image based on the first face image, the facial landmark conversion model, and the multiple SDKs to obtain a second face image processed by the multiple SDKs includes: The third facial key point is input into the facial key point conversion model to obtain the second facial key point that matches the target SDK. When the target SDK is the first SDK in the processing order, the third facial key point is the first facial key point. When the target SDK is not the first SDK in the processing order, the third facial key point is the facial key point corresponding to the fourth facial image output by the SDK preceding the target SDK. The second facial key points matching the target SDK and the target image are input to the target SDK to obtain the fifth facial image output by the target SDK. When the target SDK is the first SDK in the processing order, the target image is the first facial image. When the target SDK is not the first SDK in the processing order, the target image is the fourth facial image output by the SDK preceding the target SDK. When the target SDK is the last SDK in the processing order, the fifth facial image output by the target SDK is the second facial image after being processed by multiple SDKs.

[0010] Optionally, the step of inputting the first facial landmark into a facial landmark transformation model to obtain a second facial landmark output by the facial landmark transformation model that matches each of the plurality of SDKs includes: For each SDK, the first facial landmark and the SDK identifier corresponding to the SDK are input into the facial landmark conversion model to obtain the second facial landmark that matches the SDK, output by the facial landmark conversion model.

[0011] Optionally, before inputting the first facial landmark into the facial landmark conversion model, the method further includes: For each SDK, the SDK identifier corresponding to the SDK and the fourth facial key point corresponding to the sample face image are input into the facial key point conversion model to obtain the fifth facial key point output by the facial key point conversion model, wherein the fourth facial key point corresponds to the same key point format as the first facial key point. For each SDK, the facial landmark conversion model is trained based on the loss values ​​corresponding to the fifth facial landmark and the fourth facial landmark of the sample face image, wherein the keypoint format corresponding to the fifth facial landmark of the sample face image is the keypoint format used by the SDK.

[0012] Optionally, after obtaining the second facial landmarks output by the facial landmark transformation model that match each of the plurality of SDKs, the method further includes: For each SDK, the second facial key points matched by the SDK are input into the facial key point adjustment model to obtain the facial key points after the adjustment is performed on the second facial key points matched by the SDK according to the key point format used by the SDK.

[0013] Optionally, before determining the multiple software development kits (SDKs) for processing the first face image, the process includes: An SDK selection page is displayed, showing the processing effects corresponding to each SDK. The determination of multiple software development kits (SDKs) for processing the first face image includes: The multiple SDKs selected by the user on the SDK selection page are determined as the multiple SDKs for processing the first face image.

[0014] Optionally, the method further includes: The facial edges of the second face image are smoothed based on the boundary smoothing algorithm.

[0015] Secondly, an apparatus for processing facial images is provided, the apparatus comprising: The acquisition module is used to acquire the first face image to be processed; The extraction module is used to extract the first facial key points corresponding to the first facial image; A determination module is used to determine multiple software development kits (SDKs) for processing the first face image; The processing module is used to process the first face image based on the first face image, the face key point conversion model, and multiple SDKs to obtain a second face image processed by multiple SDKs. The face key point conversion model is used to generate second face key points that match each SDK based on the first face key points.

[0016] Optionally, the processing module is configured to: The first facial landmark is input into the facial landmark conversion model to obtain the second facial landmark output by the facial landmark conversion model that matches each of the multiple SDKs; Based on multiple SDKs and second facial key points matched with each SDK, the first face image is processed to obtain a second face image processed by multiple SDKs.

[0017] Optionally, multiple SDKs are used to process the overall image of the first face image, and the processing module is used to: The first face image is processed according to the processing order of the multiple SDKs and the second facial key points matched by each SDK. The image processed by the first SDK among the multiple SDKs is the first face image, the images processed by the other SDKs among the multiple SDKs are the processing results of the corresponding previous SDKs, and the processing result of the last SDK among the multiple SDKs is the second face image.

[0018] Optionally, multiple SDKs are used to process local images of the first face image, and the processing module is used to: Each SDK is controlled to process the first face image based on the matched second facial key points, so as to obtain a third face image after processing the first face image by each SDK; The multiple third face images are fused together to obtain the second face image.

[0019] Optionally, the plurality of SDKs correspond to a processing order, and the processing module is used for: The third facial key point is input into the facial key point conversion model to obtain the second facial key point that matches the target SDK. When the target SDK is the first SDK in the processing order, the third facial key point is the first facial key point. When the target SDK is not the first SDK in the processing order, the third facial key point is the facial key point corresponding to the fourth facial image output by the SDK preceding the target SDK. The second facial key points matching the target SDK and the target image are input to the target SDK to obtain the fifth facial image output by the target SDK. When the target SDK is the first SDK in the processing order, the target image is the first facial image. When the target SDK is not the first SDK in the processing order, the target image is the fourth facial image output by the SDK preceding the target SDK. When the target SDK is the last SDK in the processing order, the fifth facial image output by the target SDK is the second facial image after being processed by multiple SDKs.

[0020] Optionally, the processing module is configured to input the first facial key points and the SDK identifier corresponding to the SDK into the facial key point conversion model for each SDK, so as to obtain the second facial key points that match the SDK output by the facial key point conversion model.

[0021] Optionally, the device further includes a training module, which is used to input the SDK identifier corresponding to the SDK and the fourth facial key point corresponding to the sample face image into the facial key point conversion model for each SDK to obtain the fifth facial key point output by the facial key point conversion model, wherein the fourth facial key point corresponds to the same key point format as the first facial key point. For each SDK, the facial landmark conversion model is trained based on the loss values ​​corresponding to the fourth facial landmark and the fifth facial landmark of the sample face image, wherein the keypoint format corresponding to the fifth facial landmark of the sample face image is the keypoint format used by the SDK.

[0022] Optionally, the device further includes an adjustment module, used to input the second facial key points matched by the SDK into a facial key point adjustment model for each SDK, to obtain facial key points after the facial key point adjustment adjusts the second facial key points matched by the SDK according to the key point format used by the SDK.

[0023] Optionally, the device further includes a display module for displaying an SDK selection page, which shows the processing effect corresponding to each SDK; The determining module is used to determine the multiple SDKs selected by the user on the SDK selection page as multiple SDKs for processing the first face image.

[0024] Optionally, the device further includes a smoothing module for smoothing the facial edges of the second face image based on a boundary smoothing algorithm.

[0025] Thirdly, a computer device is provided, the computer device including a processor and a memory, the memory storing at least one instruction, the instruction being loaded and executed by the processor to perform the operations performed by the method for processing a face image as described in the first aspect above.

[0026] Fourthly, a computer-readable storage medium is provided, the storage medium storing at least one instruction, the instruction being loaded and executed by a processor to perform the operations performed by the method for processing a face image as described in the first aspect above.

[0027] Fifthly, a computer program product is provided, the computer program product including program code, the program code being loaded and executed by a processor to perform the operations performed by the method for processing a face image as described in the first aspect above.

[0028] The beneficial effects of the technical solution provided in this application are: The technical solution provided in this application includes a facial landmark conversion model. This model can convert first facial landmarks extracted from a first face image into second facial landmarks that are matched to each SDK. This allows multiple SDKs to process the face image based on their respective second facial landmarks, resulting in a second face image processed by multiple SDKs. Therefore, in this embodiment, multiple SDKs can be combined to process the face image, providing flexibility in using SDKs for face image processing. Attached Figure Description

[0029] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1 This is a schematic diagram of an SDK selection page provided in an embodiment of this application; Figure 2 This is a flowchart of a method for processing face images provided in an embodiment of this application; Figure 3 This is a schematic diagram of an SDK selection page provided in an embodiment of this application; Figure 4 This is a schematic diagram illustrating the use of multiple SDKs to process face images, as provided in an embodiment of this application. Figure 5 This is a schematic diagram illustrating the use of multiple SDKs to process face images, as provided in an embodiment of this application. Figure 6 This is a flowchart of a method for processing face images provided in an embodiment of this application; Figure 7 This is a structural diagram of an apparatus for processing facial images provided in an embodiment of this application; Figure 8 This is a schematic diagram of a computer device provided in an embodiment of this application; Figure 9 This is a schematic diagram of the structure of a server provided in an embodiment of this application. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0032] Video processing applications often involve applying special effects to facial images included in the video. For example, in live streaming applications, it is common to apply beauty filters to the anchor's facial image or add decorations to the facial image.

[0033] Currently, video processing applications primarily utilize third-party SDKs (Software Development Kits) to process facial images and achieve special effects. In some embodiments, video processing applications (such as short video applications) can provide multiple SDKs for users to choose from, each achieving different facial processing effects. For example, a video processing application may include multiple SDKs for beautification (hereinafter referred to as beautification SDKs), each emphasizing different beautification effects; some may focus on skin smoothing, while others may focus on face slimming. Another example is the inclusion of multiple SDKs for adding decorations (hereinafter referred to as decoration SDKs), each adding different decorations; for instance, some decoration SDKs add glasses, while others add masks. Yet another example is the inclusion of multiple SDKs for adding facial effects (hereinafter referred to as effects SDKs), each adding different effects; for example, some effects SDKs add a large head effect, while others add an aging effect.

[0034] In video processing applications, whether short video apps or live streaming apps, an SDK selection entry (face image processing entry) can be displayed on the shooting page. For example... Figure 1 As shown, after the user clicks the SDK selection entry, the video processing application displays the SDK selection page. The SDK selection page displays icons indicating the SDK processing effects; the user can then select the desired SDK to apply to the captured face image based on these icon prompts.

[0035] Currently, video processing applications only support users selecting one SDK to process facial images, and cannot overlay SDKs. Furthermore, different SDKs may use different formats for facial landmarks. Facial landmarks are sets of coordinate points used to describe the contours and features of a face, such as the corners of the eyes, the tip of the nose, and the corners of the mouth, and typically include standard formats such as 68, 83, or 106 points. This means that different SDKs may have incompatible facial landmarks. This further complicates the use of mixed SDKs. Therefore, current video processing applications still suffer from a lack of flexibility and a single, limited approach to SDK usage.

[0036] Figure 2This is a flowchart illustrating a method for processing facial images according to an embodiment of this application. In one example, the method provided in this embodiment can be executed by a terminal or a server. When the method provided in this embodiment is executed by a terminal, the method can be set in a video processing application running on the terminal. When the method provided in this embodiment is executed by a server, the service implementing the method in the server can be a background service of the video processing application. Figure 2 As shown, the method includes: Step 201: Obtain the first face image to be processed.

[0037] The first face image to be processed can be a video frame that includes a face image in the video.

[0038] When step 201 is executed by the terminal, the first face image can be a face image including the user's face captured by the terminal's camera. For example, if the terminal is running a live streaming application, the terminal can capture the streamer's face with its camera to obtain a video frame including the streamer's face, and this video frame including the streamer's face is the first face image. When step 201 is executed by the server, the first face image can be a face image including the user's face sent by the terminal to the server, such as a video frame including the streamer's face sent by the terminal to the server.

[0039] Step 202: Extract the key points of the first face corresponding to the first face image.

[0040] Among them, the face recognition model can be trained in advance using a large number of face images and can be deployed in the terminal or server.

[0041] After acquiring the first face image, first facial landmarks can be extracted from the first face image. For example, the first facial landmarks can be extracted from the first face image using a pre-set face recognition model. In this embodiment, the first facial landmark is a reference facial landmark. The reference facial landmark can be converted into a format of facial landmarks matched by the SDK. The reference facial landmark can be set by a technician. In one feasible approach, the number of facial landmarks included in the reference facial landmark is greater than or equal to the number of facial landmarks used by each SDK. For example, the reference facial landmark may be a set of 106 facial landmarks.

[0042] Step 203: Determine the multiple software development kits (SDKs) for processing the first face image.

[0043] Multiple SDKs for processing the first face image can be specified by the user. For example, in a video processing application, an SDK selection page can be displayed. The user can select multiple SDKs to use on the SDK selection page.

[0044] Figure 3 This is a schematic diagram of an SDK selection page provided in an embodiment of this application, such as... Figure 3 As shown, the SDK selection page on the terminal displays icons for multiple SDKs, along with the processing effects of each SDK. These effects can be represented by images displayed on the icons or by text. In practice, users can click on multiple SDKs they wish to apply on the SDK selection page; the SDKs selected by the user are the ones used to process the first face image.

[0045] When step 203 is executed by the terminal, the terminal can determine the multiple SDKs selected by the user on the SDK selection page and determine the SDK identifiers corresponding to the multiple SDKs. When step 203 is executed by the server, the server can receive the SDK identifiers corresponding to the multiple SDKs sent by the terminal.

[0046] Understandably, the processing effects of multiple SDKs selected by the user can be merged and displayed on the first face image after processing by multiple SDKs. In one example, the video processing application can limit the number of SDKs the user can select to avoid too many SDKs processing the first face image, thus reducing the processing effect on the first face image.

[0047] Step 204: Based on the first face image, the facial landmark transformation model, and multiple SDKs, process the first face image to obtain a second face image after processing by multiple SDKs. The facial landmark transformation model is used to generate second facial landmarks that match each SDK based on the first facial landmarks.

[0048] In one feasible approach, the facial landmark transformation model can output a second facial landmark with matching degree for each SDK at once based on the first facial landmark. The corresponding processing includes: Step 2041: Input the first facial landmark into the facial landmark conversion model to obtain the second facial landmark output by the facial landmark conversion model that matches each of the multiple SDKs.

[0049] After identifying multiple SDKs for processing face images, a pre-trained facial landmark transformation model can be used to convert the first facial landmark into a format matching the facial landmarks of each SDK. The converted first facial landmark can be called the second facial landmark. Specifically, the second facial landmark matched by the SDK, i.e., the second facial landmark, includes a number of landmarks and corresponding positional features that conform to the facial landmark rules adopted by the SDK.

[0050] In one example, for each SDK, a model can be trained to convert baseline facial landmarks into the format used by the SDK. Thus, after determining the multiple SDKs selected by the user, the facial landmark conversion model used to convert the first facial landmark can be determined based on the correspondence between SDK identifiers and facial landmark conversion models. The first facial landmark can then be input into each facial landmark conversion model, and the model outputs a second facial landmark, whose format is the same as the format used by the SDK.

[0051] In one example, for multiple SDKs in a video processing application, a facial landmark conversion model can be set up to convert a baseline facial landmark into the format used by each SDK. In this case, the input of the facial landmark conversion model can include two parts: a first facial landmark and the SDK identifier corresponding to the SDK. Thus, the facial landmark conversion model can convert the first facial landmark into the format used by the SDK based on the SDK identifier, thereby obtaining the second facial landmark corresponding to the SDK.

[0052] Of course, since the number of facial landmark formats is limited, a portion of the input to the facial landmark conversion model can be replaced by SDK identifiers with identifiers of the facial landmark formats (referred to as format identifiers). The corresponding processing can include: for each SDK, determining the format identifier corresponding to the facial landmarks used by that SDK, for example, through a recorded correspondence between SDK identifiers and format identifiers. In this case, the input to the facial landmark conversion model can include two parts: one part is the first facial landmark, and the other part is the format identifier corresponding to the facial landmarks used by the SDK. Thus, the facial landmark conversion model can convert the first facial landmark into the format of the facial landmarks used by the SDK based on the format identifier, thereby obtaining the second facial landmark corresponding to the SDK.

[0053] The facial landmark transformation model can be implemented using a fully connected neural network, a graph neural network, or a Transformer-based architecture. This application does not limit the implementation method of the facial landmark transformation model. When the facial landmark transformation model includes an SDK identifier or a format identifier, the SDK identifier or format identifier can be represented in vector form, for example, using one-hot encoding.

[0054] It is understandable that for SDKs that use the same format for facial landmarks as the first facial landmark, it is not necessary to convert the first facial landmark through a facial landmark conversion model. Instead, the first facial landmark can be used as the second facial landmark for the SDK.

[0055] In one feasible approach, to further reduce the error of the second facial landmarks output by the facial landmark transformation model, this application embodiment provides a method for reducing the error of the second facial landmarks, including: for each SDK, inputting the second facial landmarks matched by the SDK into a facial landmark adjustment model to obtain the adjusted facial landmarks after adjusting the second facial landmarks matched by the SDK according to the landmark format used by the SDK (i.e., the format of the facial landmarks).

[0056] In implementation, a facial landmark adjustment model can be trained to correct errors in facial landmarks of various formats, thereby reducing the error of the second facial landmark output by the facial landmark conversion model. The training samples for training the facial landmark adjustment model can be standard facial landmarks conforming to various formats, as well as facial landmarks obtained by perturbing these standard facial landmarks (e.g., randomly adjusting the positions of one or more landmarks).

[0057] Step 2042: Based on multiple SDKs and second facial key points matched with each SDK, process the first face image to obtain a second face image processed by multiple SDKs.

[0058] In implementation, after obtaining the second facial key points matched by each SDK, for each SDK, the second facial key points and the face image to be processed can be input into the SDK, and the SDK will process the input face image based on the input facial key points.

[0059] It is understandable that each SDK may use a different format for facial key points, and each SDK actually includes two parts: face recognition processing and face image processing. In this embodiment, facial key points are extracted for multiple SDKs using a face recognition model and converted into a key point format that matches the SDK. Therefore, in this embodiment, when processing the first face image using the SDK, the face recognition processing built into the SDK itself can be disabled, and the corresponding facial key points and the first face image can be input into the part of the SDK to implement face image processing.

[0060] Thus, in this embodiment, a face recognition model performs face recognition processing once, and then a face landmark conversion model converts the first face landmark into a second face landmark that matches each SDK. This achieves compatibility with various SDKs while avoiding face recognition processing for each SDK, thereby improving the efficiency of processing face images using multiple SDKs.

[0061] In one feasible approach, multiple SDKs can process the first face image serially. Figure 4 This is a schematic diagram illustrating the serial processing of multiple SDK first face images provided in an embodiment of this application. Figure 4 As shown, multiple SDKs are processed in a specific order, which can be randomly set. The first SDK's input can be a first face image and its matched second facial landmarks. The output of the first SDK, representing the processed first face image, can be used as input for the second SDK. The second SDK's input also includes its matched second facial landmarks, and so on. Each SDK's input includes the output of the preceding SDK and its matched second facial landmarks, until the last SDK outputs its final processed result, which is the second face image. This way, the second face image simultaneously incorporates the processing effects of multiple SDKs, providing flexibility in processing face images using the SDKs.

[0062] In one example, if the user selects multiple SDKs that process the entire first face image, then multiple SDKs can be triggered to process the first face image sequentially. Here, "processing the entire image" can refer to processing the entire face image, such as skin smoothing, skin retouching, or face slimming, covering the entire facial area. In this case, through the sequential processing of multiple SDKs, the processing effect of each SDK can be displayed in the second face image.

[0063] Furthermore, considering that the positional errors of the second facial key points matched by multiple SDKs may exist, affecting the final processing result and thus the display quality of the second facial image, a smoothing algorithm can be applied to the facial edges of the second facial image after processing the first facial image using multiple SDKs to obtain the second facial image, thereby improving the display quality of the second facial image.

[0064] In one feasible approach, multiple SDKs can process the first face image in parallel. Figure 5 This is a schematic diagram illustrating the serial-parallel processing of multiple SDK first face images provided in an embodiment of this application. For example... Figure 5 As shown, multiple SDKs can process the first face image independently. Each SDK's input includes the first face image and matched second facial landmarks. After each SDK processes the first face image, it outputs a third face image. Then, a fusion algorithm can be used to fuse these third face images from multiple SDKs to obtain a second face image processed by all SDKs. This way, the second face image incorporates the processing effects of multiple SDKs, thus providing greater flexibility in processing face images using the SDKs.

[0065] In one example, if multiple SDKs selected by the user all process local portions of a first face image, parallel processing of the first face image by multiple SDKs can be triggered. This processing of local portions could refer to processing only a portion of the face image, such as adding decorations. In this case, parallel processing by multiple SDKs allows the processing effects of each SDK to be displayed in a second face image.

[0066] In one feasible approach, multiple SDKs are assigned a processing order, and the SDKs can sequentially generate corresponding matching second facial key points for each SDK. The corresponding processing includes: Step 204a: Input the third facial landmark into the facial landmark conversion model to obtain the second facial landmark that matches the target SDK. When the target SDK is the first SDK in the processing order, the third facial landmark is the first facial landmark. When the target SDK is not the first SDK in the processing order, the third facial landmark is the facial landmark corresponding to the fourth facial image output by the SDK preceding the target SDK.

[0067] In other words, for the first SDK in the processing order, the first facial landmark can be input into the facial landmark transformation model, and the facial landmark transformation model outputs the second facial landmark matched by the first SDK. For the second and subsequent SDKs in the processing order, the facial landmarks corresponding to the fourth face image output by the previous SDK can be extracted first, and then the extracted facial landmarks can be input into the facial landmark transformation model, which outputs the second facial landmark matched by the corresponding SDK.

[0068] Step 204b: Input the second facial key points matched with the target SDK and the target image into the target SDK to obtain the fifth facial image output by the target SDK. When the target SDK is the first SDK in the processing order, the target image is the first facial image. When the target SDK is not the first SDK in the processing order, the target image is the fourth facial image output by the SDK preceding the target SDK. When the target SDK is the last SDK in the processing order, the fifth facial image output by the target SDK is the second facial image after processing by multiple SDKs.

[0069] In other words, for the first SDK in the processing order, the matched second facial landmarks obtained from the facial landmark model and the first face image can be input into the first SDK, and the first SDK outputs the processed fifth face image. For the second and subsequent SDKs in the processing order, the fourth face image output from the previous SDK and the second facial landmarks obtained from the facial landmark model that match the corresponding SDK can be input into the corresponding SDK, and the corresponding SDK outputs the processed fifth face image. The fifth face image output by the last SDK is the second face image after processing by multiple SDKs.

[0070] Figure 6 This is a schematic diagram illustrating a method for training a facial landmark transformation model according to an embodiment of this application. Figure 6 As shown, the training of the facial landmark transformation model includes: Step 601: For each SDK, input the SDK identifier corresponding to the SDK and the fourth facial key point corresponding to the sample face image into the facial key point conversion model to obtain the fifth facial key point output by the facial key point conversion model. The fourth facial key point corresponds to the key point with the same key point format as the first facial key point.

[0071] In implementation, the input to the facial landmark conversion model can include two parts: an SDK identifier (or a format identifier corresponding to the landmark format) and a baseline facial landmark. This baseline facial landmark, referred to as the fourth facial landmark during training, has the same landmark format as the first facial landmark. Correspondingly, the facial landmark conversion model can output the facial landmark corresponding to the SDK identifier (or the landmark format corresponding to the format identifier), which is the fifth facial landmark, based on the SDK identifier (or format identifier).

[0072] Step 602: For each SDK, train the facial landmark conversion model based on the loss values ​​corresponding to the fifth and fourth facial landmarks of the sample face image. The keypoint format corresponding to the fifth facial landmark of the sample face image is the keypoint format used by the SDK.

[0073] After obtaining the fifth facial landmark from the facial landmark transformation model, the loss values ​​of the fourth and fifth facial landmarks can be calculated, and then the facial landmark transformation model can be trained using the loss values.

[0074] Understandably, steps 601 and 602 above output the training process for one SDK. During training, the facial landmark conversion model can be trained using a large number of sample face images corresponding to multiple SDKs until it converges and reaches the set accuracy, thus completing the training of the facial landmark conversion model. In this way, the trained facial landmark conversion model can convert baseline facial landmarks into the format of facial landmarks used by each SDK, thereby enabling the integrated use of SDKs in video processing applications and improving the flexibility of using SDKs to process face images.

[0075] All of the above-mentioned optional technical solutions can be combined in any way to form optional embodiments of this disclosure, and will not be described in detail here.

[0076] Based on the same inventive concept, embodiments of this application also provide an apparatus for processing facial images. Figure 7 This is a schematic diagram of the apparatus structure for processing face images provided in an embodiment of this application, as shown below. Figure 7 As shown, the device includes: The acquisition module 710 is used to acquire the first face image to be processed; Extraction module 720 is used to extract the first facial key points corresponding to the first facial image; The determination module 730 is used to determine multiple software development kits (SDKs) for processing the first face image; The processing module 740 is used to process the first face image based on the first face image, the face key point conversion model, and multiple SDKs to obtain a second face image processed by multiple SDKs, wherein the face key point conversion model is used to generate second face key points that match each SDK based on the first face key points.

[0077] Optionally, the processing module 740 is configured to: The first facial landmark is input into the facial landmark conversion model to obtain the second facial landmark output by the facial landmark conversion model that matches each of the multiple SDKs; Based on multiple SDKs and second facial key points matched with each SDK, the first face image is processed to obtain a second face image processed by multiple SDKs.

[0078] Optionally, multiple SDKs are used to process the overall image of the first face image, and the processing module 740 is used to: The first face image is processed according to the processing order of the multiple SDKs and the second facial key points matched by each SDK. The image processed by the first SDK among the multiple SDKs is the first face image, the images processed by the other SDKs among the multiple SDKs are the processing results of the corresponding previous SDKs, and the processing result of the last SDK among the multiple SDKs is the second face image.

[0079] Optionally, multiple SDKs are used to process local images of the first face image, and the processing module 740 is used to: Each SDK is controlled to process the first face image based on the matched second facial key points, so as to obtain a third face image after processing the first face image by each SDK; The multiple third face images are fused together to obtain the second face image.

[0080] Optionally, the plurality of SDKs correspond to a processing order, and the processing module 740 is used for: The third facial key point is input into the facial key point conversion model to obtain the second facial key point that matches the target SDK. When the target SDK is the first SDK in the processing order, the third facial key point is the first facial key point. When the target SDK is not the first SDK in the processing order, the third facial key point is the facial key point corresponding to the fourth facial image output by the SDK preceding the target SDK. The second facial key points matching the target SDK and the target image are input to the target SDK to obtain the fifth facial image output by the target SDK. When the target SDK is the first SDK in the processing order, the target image is the first facial image. When the target SDK is not the first SDK in the processing order, the target image is the fourth facial image output by the SDK preceding the target SDK. When the target SDK is the last SDK in the processing order, the fifth facial image output by the target SDK is the second facial image after being processed by multiple SDKs.

[0081] Optionally, the processing module 740 is configured to input the first facial key points and the SDK identifier corresponding to the SDK into the facial key point conversion model for each SDK, so as to obtain the second facial key points that match the SDK output by the facial key point conversion model.

[0082] Optionally, the device further includes a training module, which is used to input the SDK identifier corresponding to the SDK and the fourth facial key point corresponding to the sample face image into the facial key point conversion model for each SDK to obtain the fifth facial key point output by the facial key point conversion model, wherein the fourth facial key point corresponds to the same key point format as the first facial key point. For each SDK, the facial landmark conversion model is trained based on the loss values ​​corresponding to the fourth facial landmark and the fifth facial landmark of the sample face image, wherein the keypoint format corresponding to the fifth facial landmark of the sample face image is the keypoint format used by the SDK.

[0083] Optionally, the device further includes an adjustment module, used to input the second facial key points matched by the SDK into a facial key point adjustment model for each SDK, to obtain facial key points after the facial key point adjustment adjusts the second facial key points matched by the SDK according to the key point format used by the SDK.

[0084] Optionally, the device further includes a display module for displaying an SDK selection page, which shows the processing effect corresponding to each SDK; The determining module 730 is used to determine the multiple SDKs selected by the user on the SDK selection page as multiple SDKs for processing the first face image.

[0085] Optionally, the device further includes a smoothing module for smoothing the facial edges of the second face image based on a boundary smoothing algorithm.

[0086] It should be noted that the facial image processing apparatus provided in the above embodiments is only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. Furthermore, the facial image processing apparatus and the facial image processing method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process is detailed in the method embodiments, which will not be repeated here.

[0087] Figure 8This illustration shows a schematic diagram of a computer device provided in an exemplary embodiment of this application. The computer device can be the terminal involved in the above embodiments, and can be referred to as electronic device 800. Electronic device 800 can be a portable mobile terminal, such as a smartphone, tablet computer, MP3 player (Moving Picture Experts Group Audio Layer III), MP4 player (Moving Picture Experts Group Audio Layer IV), laptop computer, or desktop computer. Electronic device 800 may also be referred to as user equipment, portable terminal, laptop terminal, desktop terminal, or other names.

[0088] Typically, electronic device 800 includes a processor 801 and a memory 802.

[0089] Processor 801 may include one or more processing cores, such as a quad-core processor or an octa-core processor. Processor 801 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). Processor 801 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 801 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 801 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0090] The memory 802 may include one or more computer-readable storage media, which may be non-transitory. The memory 802 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 802 are used to store at least one instruction, which is executed by the processor 801 to implement the method for processing face images provided in the method embodiments of this application.

[0091] In some embodiments, the electronic device 800 may optionally include a peripheral device interface 803 and at least one peripheral device. The processor 801, memory 802, and peripheral device interface 803 can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface 803 via a bus, signal line, or circuit board. Specifically, the peripheral device includes at least one of the following: a radio frequency circuit 804, a display screen 805, a camera assembly 806, an audio circuit 807, a positioning assembly 808, and a power supply 809.

[0092] Peripheral device interface 803 can be used to connect at least one I / O (Input / Output) related peripheral device to processor 801 and memory 802. In some embodiments, processor 801, memory 802 and peripheral device interface 803 are integrated on the same chip or circuit board; in some other embodiments, any one or two of processor 801, memory 802 and peripheral device interface 803 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0093] The radio frequency (RF) circuit 804 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF circuit 804 communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 804 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals back into electrical signals. Optionally, the RF circuit 804 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, etc. The RF circuit 804 can communicate with other terminals through at least one wireless communication protocol. This wireless communication protocol includes, but is not limited to: the World Wide Web, metropolitan area networks, intranets, various generations of mobile communication networks (2G, 3G, 4G, and 5G), and wireless local area networks. In some embodiments, the RF circuit 804 may also include circuitry related to NFC (Near Field Communication), which is not limited in this application.

[0094] Display screen 805 is used to display a UI (User Interface). This UI may include graphics, text, icons, videos, and any combination thereof. When display screen 805 is a touch display screen, it also has the ability to collect touch signals on or above its surface. These touch signals can be input as control signals to processor 801 for processing. In this case, display screen 805 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, there may be one display screen 805, disposed on the front panel of electronic device 800; in other embodiments, there may be at least two display screens, disposed on different surfaces of electronic device 800 or in a folded design; in still other embodiments, display screen 805 may be a flexible display screen, disposed on a curved or folded surface of electronic device 800. Furthermore, display screen 805 may be configured as a non-rectangular irregular shape, i.e., a non-rectangular screen. Display screen 805 may be made of materials such as LCD (Liquid Crystal Display) or OLED (Organic Light-Emitting Diode).

[0095] The camera assembly 806 is used to acquire images or videos. Optionally, the camera assembly 806 includes a front-facing camera and a rear-facing camera. Typically, the front-facing camera is located on the front panel of the terminal, and the rear-facing camera is located on the back of the terminal. In some embodiments, there are at least two rear-facing cameras, which are any one of a main camera, a depth-sensing camera, a wide-angle camera, and a telephoto camera, to achieve background blurring by fusion of the main camera and the depth-sensing camera, panoramic shooting by fusion of the main camera and the wide-angle camera, VR (Virtual Reality) shooting, or other fusion shooting functions. In some embodiments, the camera assembly 806 may also include a flash. The flash can be a single-color temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm-light flash and a cool-light flash, which can be used for light compensation at different color temperatures.

[0096] The audio circuit 807 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, converting the sound waves into electrical signals that are input to the processor 801 for processing, or input to the radio frequency circuit 804 for voice communication. For stereo sound acquisition or noise reduction purposes, multiple microphones may be used, each located in a different part of the electronic device 800. The microphone may also be an array microphone or an omnidirectional microphone. The speaker is used to convert the electrical signals from the processor 801 or the radio frequency circuit 804 into sound waves. The speaker may be a conventional diaphragm speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can convert electrical signals not only into audible sound waves but also into inaudible sound waves for purposes such as distance measurement. In some embodiments, the audio circuit 807 may also include a headphone jack.

[0097] The positioning component 808 is used to locate the current geographic location of the electronic device 800 in order to enable navigation or LBS (Location Based Service).

[0098] Power supply 809 is used to supply power to various components in electronic device 800. Power supply 809 can be alternating current, direct current, a disposable battery, or a rechargeable battery. When power supply 809 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery that is charged via a wired line, while a wireless rechargeable battery is a battery that is charged via a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0099] In some embodiments, the electronic device 800 further includes one or more sensors 810. The one or more sensors 810 include, but are not limited to: an accelerometer 811, a gyroscope 812, a pressure sensor 813, a fingerprint sensor 814, an optical sensor 815, and a proximity sensor 816.

[0100] Accelerometer 811 can detect the magnitude of acceleration on the three coordinate axes of a coordinate system established by electronic device 800. For example, accelerometer 811 can be used to detect the components of gravitational acceleration on the three coordinate axes. Processor 801 can control display screen 805 to display the user interface in either a landscape or portrait view based on the gravitational acceleration signal acquired by accelerometer 811. Accelerometer 811 can also be used for games or for acquiring user motion data.

[0101] The gyroscope sensor 812 can detect the orientation and rotation angle of the electronic device 800. The gyroscope sensor 812, in conjunction with the accelerometer sensor 811, can collect 3D motion data from the user on the electronic device 800. Based on the data collected by the gyroscope sensor 812, the processor 801 can perform the following functions: motion sensing (e.g., changing the UI based on the user's tilt), image stabilization during shooting, game control, and inertial navigation.

[0102] The pressure sensor 813 can be disposed on the side bezel of the electronic device 800 and / or on the lower layer of the display screen 805. When the pressure sensor 813 is disposed on the side bezel of the electronic device 800, it can detect the user's grip signal on the electronic device 800, and the processor 801 can perform left / right hand recognition or quick operation based on the grip signal collected by the pressure sensor 813. When the pressure sensor 813 is disposed on the lower layer of the display screen 805, the processor 801 can control the operable controls on the UI interface based on the user's pressure operation on the display screen 805. The operable controls include at least one of button controls, scroll bar controls, icon controls, and menu controls.

[0103] The fingerprint sensor 814 is used to collect a user's fingerprint. The processor 801 identifies the user based on the fingerprint collected by the fingerprint sensor 814, or vice versa. When the user's identity is verified as trusted, the processor 801 authorizes the user to perform relevant sensitive operations, including unlocking the screen, viewing encrypted information, downloading software, making payments, and changing settings. The fingerprint sensor 814 can be located on the front, back, or side of the electronic device 800. When the electronic device 800 has a physical button or manufacturer logo, the fingerprint sensor 814 can be integrated with the physical button or manufacturer logo.

[0104] An optical sensor 815 is used to collect ambient light intensity. In one embodiment, the processor 801 can control the display brightness of the display screen 805 based on the ambient light intensity collected by the optical sensor 815. Specifically, when the ambient light intensity is high, the display brightness of the display screen 805 is increased; when the ambient light intensity is low, the display brightness of the display screen 805 is decreased. In another embodiment, the processor 801 can also dynamically adjust the shooting parameters of the camera assembly 806 based on the ambient light intensity collected by the optical sensor 815.

[0105] A proximity sensor 816, also known as a distance sensor, is typically located on the front panel of an electronic device 800. The proximity sensor 816 is used to detect the distance between the user and the front of the electronic device 800. In one embodiment, when the proximity sensor 816 detects that the distance between the user and the front of the electronic device 800 is gradually decreasing, the processor 801 controls the display screen 805 to switch from a screen-on state to a screen-off state; when the proximity sensor 816 detects that the distance between the user and the front of the electronic device 800 is gradually increasing, the processor 801 controls the display screen 805 to switch from a screen-off state to a screen-on state.

[0106] Those skilled in the art will understand that Figure 8 The structure shown does not constitute a limitation on the electronic device 800, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0107] Figure 9 This is a schematic diagram of a server structure provided in an embodiment of this application. The server 900 can vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 901 and one or more memories 902. The memories 902 store at least one instruction, which is loaded and executed by the processors 901 to implement the methods provided in the various method embodiments described above. Of course, the server may also have wired or wireless network interfaces, a keyboard, and input / output interfaces for input and output. The server may also include other components for implementing device functions, which will not be elaborated upon here.

[0108] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including instructions that can be executed by a processor in a terminal to perform the method for processing facial images in the above embodiments. This computer-readable storage medium can be non-transitory. For example, the computer-readable storage medium can be ROM (Read-Only Memory), RAM (Random Access Memory), CD-ROM (CompactDisc Read-Only Memory), magnetic tape, floppy disk, and optical data storage devices, etc.

[0109] In an exemplary embodiment, a computer program product is also provided, which includes program code that can be executed by a processor in a computer device to perform the method for processing facial images described above.

[0110] It should be noted that all information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals (including but not limited to signals transmitted between the user terminal and other devices) involved in this application have been authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the facial images involved in this application were obtained with full authorization.

[0111] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0112] It should be understood that "multiple" as used herein refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. Furthermore, the step numbers described herein are merely illustrative of one possible execution order. In some other embodiments, the steps may not be executed in numerical order, such as two steps with different numbers being executed simultaneously, or two steps with different numbers being executed in the reverse order of the illustration. This application does not limit this.

[0113] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for processing facial images, characterized in that, The method includes: Obtain the first face image to be processed; Extract the first facial key points corresponding to the first face image; Identify multiple software development kits (SDKs) for processing the first face image; Based on the first face image, the facial landmark conversion model, and multiple SDKs, the first face image is processed to obtain a second face image processed by multiple SDKs. The facial landmark conversion model is used to generate second facial landmarks that match each SDK based on the first facial landmarks.

2. The method according to claim 1, characterized in that, The step of processing the first face image based on the first face image, the facial landmark conversion model, and multiple SDKs to obtain a second face image processed by the multiple SDKs includes: The first facial landmark is input into the facial landmark conversion model to obtain the second facial landmark output by the facial landmark conversion model that matches each of the multiple SDKs; Based on multiple SDKs and second facial key points matched with each SDK, the first face image is processed to obtain a second face image processed by multiple SDKs.

3. The method according to claim 2, characterized in that, Multiple SDKs are used to process the overall image of the first face image. The process of processing the first face image based on the multiple SDKs and second facial key points matched with each SDK to obtain a second face image processed by the multiple SDKs includes: The first face image is processed according to the processing order of the multiple SDKs and the second facial key points matched by each SDK. The image processed by the first SDK among the multiple SDKs is the first face image, the images processed by the other SDKs among the multiple SDKs are the processing results of the corresponding previous SDKs, and the processing result of the last SDK among the multiple SDKs is the second face image.

4. The method according to claim 2, characterized in that, Multiple SDKs are used to process local images of the first face image. The process of processing the first face image based on the multiple SDKs and second facial key points matched with each SDK to obtain a second face image processed by the multiple SDKs includes: Each SDK is controlled to process the first face image based on the matched second facial key points, so as to obtain a third face image after processing the first face image by each SDK; The multiple third face images are fused together to obtain the second face image.

5. The method according to claim 1, characterized in that, The multiple SDKs correspond to a processing order. The step of processing the first face image based on the first face image, the facial landmark conversion model, and the multiple SDKs to obtain a second face image processed by the multiple SDKs includes: The third facial key point is input into the facial key point conversion model to obtain the second facial key point that matches the target SDK. When the target SDK is the first SDK in the processing order, the third facial key point is the first facial key point. When the target SDK is not the first SDK in the processing order, the third facial key point is the facial key point corresponding to the fourth facial image output by the SDK preceding the target SDK. The second facial key points matching the target SDK and the target image are input to the target SDK to obtain the fifth facial image output by the target SDK. When the target SDK is the first SDK in the processing order, the target image is the first facial image. When the target SDK is not the first SDK in the processing order, the target image is the fourth facial image output by the SDK preceding the target SDK. When the target SDK is the last SDK in the processing order, the fifth facial image output by the target SDK is the second facial image after being processed by multiple SDKs.

6. The method according to any one of claims 2 to 4, characterized in that, The step of inputting the first facial landmark into a facial landmark conversion model to obtain the second facial landmark output by the facial landmark conversion model that matches each of the multiple SDKs includes: For each SDK, the first facial landmark and the SDK identifier corresponding to the SDK are input into the facial landmark conversion model to obtain the second facial landmark that matches the SDK, output by the facial landmark conversion model.

7. The method according to claim 6, characterized in that, Before inputting the first facial landmark into the facial landmark conversion model, the method further includes: For each SDK, the SDK identifier corresponding to the SDK and the fourth facial key point corresponding to the sample face image are input into the facial key point conversion model to obtain the fifth facial key point output by the facial key point conversion model, wherein the fourth facial key point corresponds to the same key point format as the first facial key point. For each SDK, the facial landmark conversion model is trained based on the loss values ​​corresponding to the fifth facial landmark and the fourth facial landmark of the sample face image, wherein the keypoint format corresponding to the fifth facial landmark of the sample face image is the keypoint format used by the SDK.

8. The method according to any one of claims 2 to 4, characterized in that, After obtaining the second facial landmarks output by the facial landmark transformation model that match each of the multiple SDKs, the method further includes: For each SDK, the second facial key points matched by the SDK are input into the facial key point adjustment model to obtain the facial key points after the adjustment is performed on the second facial key points matched by the SDK according to the key point format used by the SDK.

9. The method according to any one of claims 2 to 4, characterized in that, Before determining the multiple software development kits (SDKs) for processing the first face image, the process includes: An SDK selection page is displayed, showing the processing effects corresponding to each SDK. The determination of multiple software development kits (SDKs) for processing the first face image includes: The multiple SDKs selected by the user on the SDK selection page are determined as the multiple SDKs for processing the first face image.

10. The method according to claim 3, characterized in that, The method further includes: The facial edges of the second face image are smoothed based on the boundary smoothing algorithm.

Images