Face model processing method and device, electronic equipment and storage medium

Abstract

The application discloses a face model processing method and device, electronic equipment and a storage medium, and belongs to the technical field of computers. The method comprises the following steps: acquiring a reference face image, wherein the reference face image comprises a wrinkle image region; performing block processing on the wrinkle image region to obtain a plurality of wrinkle image blocks; determining a target wrinkle map based on the plurality of wrinkle image blocks; determining a model region in a face model to be processed, wherein the model region is a region corresponding to the target wrinkle map on the face model to be processed; and fusing the target wrinkle map onto the model region to obtain a face model with wrinkles. The application determines the target wrinkle map based on the wrinkle image region of the reference face image, reduces the time required for obtaining the wrinkle map, thereby reducing the time required for obtaining the face model with wrinkles, and improving the efficiency of obtaining the face model with wrinkles.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, apparatus, electronic device, and storage medium for processing facial models. Background Technology

[0002] With the continuous development of computer technology, the demands for facial models in industries such as film and games are increasing daily. By processing the facial model to be processed, a facial model with wrinkles can be obtained, which can improve the realism of the facial model.

[0003] In related technologies, wrinkles can be manually drawn on a texture map to obtain a wrinkle texture map. This wrinkle texture map is then fused to the corresponding model area in the facial model to be processed, resulting in a facial model with wrinkles. However, manually drawing wrinkles takes a long time, which in turn makes obtaining a wrinkle texture map inefficient. Summary of the Invention

[0004] This application provides a method, apparatus, electronic device, and storage medium for processing facial models, which can be used to solve the problem of low efficiency in obtaining facial models with wrinkles due to the long time required to obtain wrinkle maps in related technologies. The technical solution includes the following contents.

[0005] On the one hand, a method for processing facial models is provided, the method comprising:

[0006] Obtain a reference facial image, which includes a wrinkle image region;

[0007] The wrinkle image region is divided into blocks to obtain multiple wrinkle image blocks;

[0008] The target wrinkle map is determined based on the multiple wrinkle image blocks;

[0009] In the facial model to be processed, a model region is determined, which is the region on the facial model to be processed that corresponds to the target wrinkle texture.

[0010] The target wrinkle texture is fused onto the model area to obtain a facial model with wrinkles.

[0011] On the other hand, a facial model processing apparatus is provided, the apparatus comprising:

[0012] The acquisition module is used to acquire a reference facial image, which includes a wrinkle image region;

[0013] The segmentation module is used to segment the wrinkle image region into multiple wrinkle image blocks.

[0014] The determination module is used to determine the target wrinkle map based on the plurality of wrinkle image blocks;

[0015] The determining module is used to determine a model region in the face model to be processed, wherein the model region is the region on the face model to be processed that corresponds to the target wrinkle texture.

[0016] The fusion module is used to fuse the target wrinkle texture onto the model area to obtain a facial model with wrinkles.

[0017] In one possible implementation, the device further includes:

[0018] The filtering module is used to filter the reference facial image to obtain the filtering result;

[0019] The determining module is further configured to determine, based on the filtering result, the probability that each pixel in the reference facial image belongs to the pixel corresponding to the wrinkle; and to determine the wrinkle image region based on the probability that each pixel in the reference facial image belongs to the pixel corresponding to the wrinkle.

[0020] In one possible implementation, the filtering module is used to perform differential filtering on the reference facial image to obtain a differential filtering result; and to perform average filtering on the differential filtering result to obtain an average filtering result.

[0021] In one possible implementation, the segmentation module is used to segment the wrinkle image region based on multiple segmentation windows to obtain initial image blocks; determine rotation parameters of each segmentation window based on each initial image block; rotate each segmentation window based on the rotation parameters to obtain rotated segmentation windows; and segment the wrinkle image region based on the rotated segmentation windows to obtain wrinkle image blocks.

[0022] In one possible implementation, the segmentation module is configured to, for any initial image block, determine the gradient of each pixel in the horizontal direction and the gradient of each pixel in the vertical direction based on the pixel information of each pixel in the initial image block; and determine the rotation parameters of the segmented window corresponding to the initial image block based on the gradient of each pixel in the horizontal direction and the gradient of each pixel in the vertical direction.

[0023] In one possible implementation, the rotation parameter includes a rotation angle;

[0024] The segmentation module is used to determine the ratio between a first cumulative value and a second cumulative value, wherein the first cumulative value is the sum of the absolute values ​​of the gradients of each pixel in the horizontal direction in any initial image block, and the second cumulative value is the sum of the absolute values ​​of the gradients of each pixel in the vertical direction in any initial image block; and to determine the rotation angle of the segmentation window corresponding to the any initial image block based on the ratio.

[0025] In one possible implementation, the rotation parameter includes the rotation direction;

[0026] The segmentation module is used to determine the maximum value corresponding to any pixel in any initial image block, wherein the maximum value corresponding to any pixel is the maximum value between the absolute value of the gradient of the pixel in the horizontal direction and the absolute value of the gradient of the pixel in the vertical direction; and to determine the rotation direction of the segmentation window corresponding to the initial image block based on the maximum values ​​corresponding to each pixel in the initial image block.

[0027] In one possible implementation, the determining module is configured to, for any wrinkle image patch, determine the features of the texture image patch corresponding to the wrinkle image patch based on the wrinkle image patch and the feature processing model; determine the target displacement map based on the features of the texture image patches corresponding to each wrinkle image patch; and convert the target displacement map into the target wrinkle map.

[0028] In one possible implementation, the feature processing model includes at least one cascaded random fern classifier;

[0029] The determining module is used to obtain a first initial feature, which is input information set for the feature processing model; input the first initial feature into the at least one cascaded random fern classifier, and control the at least one cascaded random fern classifier to perform multiple corrections on the first initial feature based on any wrinkle image patch to obtain the feature of the texture image patch corresponding to any wrinkle image patch.

[0030] In one possible implementation, the acquisition module is further configured to acquire a sample face image and a sample permutation map corresponding to the sample face image, wherein the sample face image includes a sample image region where wrinkles are located, and the sample permutation map includes a sample map region where wrinkles are located.

[0031] The segmentation module is further configured to segment the sample image region into blocks to obtain each first image block; and to segment the sample texture region into blocks to obtain each second image block.

[0032] The acquisition module is further configured to train the neural network model based on the first image blocks and the second image blocks to obtain the feature processing model.

[0033] In one possible implementation, one first image block corresponds to one second image block;

[0034] The acquisition module is used to acquire a second initial feature, which is input information set for the neural network model; for any first image block, the neural network model is controlled to modify the second initial feature according to the first image block to obtain a first feature of the second image block corresponding to the first image block; the second feature of each second image block is extracted; based on the first feature and the second feature of each second image block, the neural network model is adjusted to obtain the feature processing model.

[0035] In one possible implementation, the neural network model includes at least one cascaded initial classifier;

[0036] The acquisition module is used to input the second initial feature into the at least one cascaded initial classifier to obtain the output information of each initial classifier. The output information of the previous initial classifier in the at least one cascaded initial classifier is the input information of the next initial classifier, and the output information of the last initial classifier is the first feature of the second image block corresponding to any first image block.

[0037] The acquisition module is used to determine the loss value of any initial classifier among the at least one cascaded initial classifiers, based on the input information of the initial classifier, the second features of each second image patch, and the output information of the initial classifier; and to adjust the neural network model based on the loss values ​​of each initial classifier among the at least one cascaded initial classifiers to obtain the feature processing model.

[0038] On the other hand, an electronic device is provided, comprising a processor and a memory, wherein the memory stores at least one computer program, which is loaded and executed by the processor to enable the electronic device to implement any of the above-described methods for processing facial models.

[0039] On the other hand, a computer-readable storage medium is also provided, wherein at least one computer program is stored therein, the at least one computer program being loaded and executed by a processor to enable an electronic device to implement any of the above-described methods for processing facial models.

[0040] On the other hand, a computer program or computer program product is also provided, wherein at least one computer program is stored in the computer program or computer program product, and the at least one computer program is loaded and executed by a processor to enable an electronic device to implement any of the above-described methods for processing facial models.

[0041] The technical solution provided in this application brings at least the following beneficial effects:

[0042] The technical solution provided in this application involves segmenting the wrinkle image region of a reference facial image into multiple wrinkle image blocks, and determining a target wrinkle texture based on these multiple wrinkle image blocks. This reduces the time required to acquire the wrinkle texture and improves the efficiency of wrinkle texture acquisition. Since the target wrinkle texture is fused to the model region corresponding to the target wrinkle texture, a facial model with wrinkles is obtained. Therefore, the time required to acquire a facial model with wrinkles is reduced, the efficiency of acquiring a facial model with wrinkles is improved, and the processing efficiency of the facial model is also improved. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 This is a schematic diagram illustrating the implementation environment of a facial model processing method provided in this application embodiment;

[0045] Figure 2 This is a flowchart of a facial model processing method provided in an embodiment of this application;

[0046] Figure 3 This is a schematic diagram illustrating a process for determining wrinkle image blocks provided in an embodiment of this application;

[0047] Figure 4 This is a schematic diagram illustrating the acquisition of an image block set provided in an embodiment of this application;

[0048] Figure 5 This is a schematic diagram illustrating the generation of a first feature of a second image according to an embodiment of this application;

[0049] Figure 6 This is a schematic diagram illustrating the deformation of a facial model according to an embodiment of this application;

[0050] Figure 7 This is a schematic diagram of a process for determining a wrinkle region provided in an embodiment of this application;

[0051] Figure 8 This is a schematic diagram of an original image and a corresponding facial model with wrinkles provided in an embodiment of this application;

[0052] Figure 9 This is a schematic diagram of the structure of a facial model processing device provided in an embodiment of this application;

[0053] Figure 10 This is a schematic diagram of the structure of a terminal device provided in an embodiment of this application;

[0054] Figure 11 This is a schematic diagram of the structure of a server provided in an embodiment of this application. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0056] Figure 1 This is a schematic diagram illustrating the implementation environment of a facial model processing method provided in this application embodiment, such as... Figure 1 As shown, the implementation environment includes a terminal device 101 and a server 102. The facial model processing method in this embodiment can be executed by the terminal device 101, the server 102, or jointly by both.

[0057] Terminal device 101 can be a smartphone, game console, desktop computer, tablet computer, laptop computer, smart TV, smart in-vehicle device, smart voice interaction device, smart home appliance, etc. Server 102 can be a single server, a server cluster consisting of multiple servers, or any of the following: cloud computing platform and virtualization center. This application embodiment does not limit this. Server 102 can communicate with terminal device 101 via a wired network or wireless network. Server 102 can have functions such as data processing, data storage, and data transmission and reception. This application embodiment does not limit this. The number of terminal devices 101 and servers 102 is not limited and can be one or more.

[0058] The facial model processing method provided in this application is based on artificial intelligence (AI) technology. AI is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results. In other words, AI is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to have perception, reasoning, and decision-making functions.

[0059] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, as well as machine learning / deep learning, autonomous driving, and intelligent transportation.

[0060] With the research and advancement of artificial intelligence (AI) technology, AI is being studied and applied in various fields, such as smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, autonomous driving, drones, robots, smart healthcare, smart customer service, vehicle networking, and intelligent transportation. It is believed that with the development of technology, AI will be applied in more fields and play an increasingly important role.

[0061] Based on the above implementation environment, this application provides a method for processing facial models, in order to... Figure 2 The flowchart shown in this embodiment of the present application illustrates a method for processing a facial model. This method can be implemented by... Figure 1 The method can be executed by either terminal device 101 or server 102, or jointly by both. For ease of description, the terminal device 101 or server 102 executing the facial model processing method in this embodiment is referred to as an electronic device, and this method can be executed by an electronic device. Figure 2 As shown, the method includes the following steps.

[0062] Step 201: Obtain a reference facial image, which includes a wrinkle image region.

[0063] In this embodiment, the reference facial image is an image containing the face of any living being, including but not limited to humans, dogs, and cats. The reference facial image includes information such as the texture and color of the living being's face. Optionally, the reference facial image is the acquired original image; for example, a facial image taken by a user or an image crawled from the internet can be used as the reference facial image. Optionally, the acquired original image is fitted with a face model to obtain the reference facial image. This can be done using a face model fitting library (e.g., the EOS face model fitting library) to fit the face of the living being in the original image, thereby obtaining the reference facial image.

[0064] In this embodiment, the reference facial image includes an image region where wrinkles are located, referred to as the wrinkle image region. It is necessary to determine the wrinkle image region in the reference facial image to divide the wrinkle image region into multiple wrinkle image blocks, and to determine the target wrinkle texture based on each wrinkle image block. Step 202 below describes the process of dividing the wrinkle image region into multiple wrinkle image blocks, and step 203 below describes the process of determining the target wrinkle texture based on each wrinkle image block; these details will not be repeated here.

[0065] Optionally, after obtaining the reference facial image, the method further includes: filtering the reference facial image to obtain a filtering result; determining the probability that each pixel in the reference facial image belongs to the pixel corresponding to the wrinkle based on the filtering result; and determining the wrinkle image region based on the probability that each pixel in the reference facial image belongs to the pixel corresponding to the wrinkle.

[0066] In this embodiment, a filtering result is obtained by filtering a reference facial image. This filtering result includes the probability that each pixel in the reference facial image belongs to a pixel corresponding to a wrinkle. This embodiment does not limit the filtering method.

[0067] Typically, a reference facial image contains not only wrinkles but also other information besides wrinkles, such as large pores and melanin. In this embodiment, the other information besides wrinkles in the reference facial image is considered noise. By filtering the reference facial image, the noise can be removed, revealing the wrinkles and thus accurately identifying the wrinkle image region within the reference facial image.

[0068] In one possible implementation, filtering the reference facial image to obtain a filtering result includes: performing differential filtering on the reference facial image to obtain a differential filtering result; and performing average filtering on the differential filtering result to obtain an average filtering result.

[0069] This application does not limit the processing method of differential filtering on the reference facial image. For example, a Gaussian difference filter is used to perform Gaussian difference filtering on the reference facial image. In this case, the differential filtering result is the Gaussian difference filtering result. The Gaussian difference filtering result is obtained as follows: the reference facial image is subjected to Gaussian blurring based on two different standard deviations to obtain the processing results corresponding to each standard deviation. The processing results corresponding to each standard deviation are subtracted to obtain the Gaussian difference filtering result.

[0070] Optionally, when performing Gaussian blur processing on a reference facial image based on a standard deviation, the Gaussian function can be determined based on this standard deviation. The Gaussian function is a normal distribution function. For any pixel in the reference facial image, a specified region centered on that pixel is determined. Weights are assigned to each pixel within this specified region based on the Gaussian function. A weighted sum is then determined based on the pixel information and weights of each pixel within the specified region. This weighted sum is used as the Gaussian blurred value corresponding to that pixel. In this way, the Gaussian blurred values ​​corresponding to each pixel in the reference facial image can be determined, where the processing result corresponding to the standard deviation includes the Gaussian blurred values ​​corresponding to each pixel in the reference facial image.

[0071] Since applying Gaussian blurring to a reference facial image based on a single standard deviation can suppress high-frequency components, applying Gaussian blurring to the reference facial image using two different standard deviations can suppress high-frequency components to varying degrees. By subtracting the processing results corresponding to the two different standard deviations, the high-frequency components in the reference facial image can be extracted, i.e., wrinkles can be extracted. Therefore, the Gaussian difference filtering result can characterize wrinkles in a reference facial image.

[0072] In application, differential filtering is not limited to Gaussian difference filtering. It can also utilize a differential filter, performing differential filtering based on the pixel values ​​of each pixel in the reference facial image. For example, first perform Gaussian filtering on the reference facial image, and then perform differential filtering based on the pixel values ​​of each pixel in the Gaussian-filtered reference facial image to obtain the differential filtering result.

[0073] It should be noted that differential filtering of the reference facial image introduces some noise. Taking Gaussian difference filtering as an example, Gaussian difference filters are commonly used for edge extraction, and this embodiment can use a Gaussian difference filter to process the reference facial image. While Gaussian difference filters can effectively identify brightness variations in facial images, they also introduce noise, potentially filtering out even slightly larger pores. Therefore, denoising the differential filtering result is necessary to remove noise information representing the reference facial image while retaining wrinkle information, thus improving the expressiveness of wrinkles in the facial model.

[0074] In this embodiment, an average filter is used to perform average filtering on the differential filtering result to reduce noise and obtain an average filtering result. The average filtering result is obtained as follows: the differential filtering result is convolved to obtain a convolution result. For any pixel in the convolution result, a pixel region centered on the position of that pixel is determined. The average value of the pixel information of each pixel in the pixel region is calculated. This average value is used as the value after average filtering of that pixel. This value can characterize the probability that the pixel in the reference facial image belongs to the pixel corresponding to a wrinkle.

[0075] This method achieves average filtering of each pixel in the differential filtering result, resulting in an average filtering result. This average filtering result includes the values ​​of each pixel after average filtering, and also includes the probability that each pixel in the reference facial image belongs to the pixel corresponding to the wrinkle.

[0076] Then, for any pixel in the reference facial image, if the probability that the pixel belongs to a pixel corresponding to a wrinkle is greater than a probability threshold, then the pixel is determined to belong to the wrinkle image region. If the probability that the pixel belongs to a pixel corresponding to a wrinkle is not greater than the probability threshold, then the pixel is determined not to belong to the wrinkle image region. In this way, the wrinkle image region in the reference facial image can be determined.

[0077] Step 202: Divide the wrinkle image region into blocks to obtain multiple wrinkle image blocks.

[0078] Optionally, a sliding block window can be used to divide the wrinkle image region into blocks. Each time the block window slides to a position, it can divide the wrinkle image region into a single wrinkle image block, resulting in multiple wrinkle image blocks. In this case, all wrinkle image blocks are of the same size. Alternatively, multiple block windows at different positions can be used to divide the wrinkle image region into blocks, with each block window dividing the wrinkle image region into a single wrinkle image block.

[0079] For ease of description, in this embodiment of the application, a sliding block window is used to represent the block window at each different position. That is, this embodiment of the application divides the wrinkle image region into blocks based on multiple block windows at different positions, so that each block window divides the wrinkle image region into a wrinkle image block. Any two block windows can have the same size (i.e., any two wrinkle image blocks can have the same size) or different sizes (i.e., any two wrinkle image blocks can have different sizes).

[0080] By dividing the wrinkle image region into multiple wrinkle image patches, subsequent determination of the target wrinkle map based on these patches involves processing fine-grained wrinkle image patches rather than coarse-grained wrinkle image regions. Processing these fine-grained wrinkle image patches improves processing accuracy, thereby enhancing the precision of the target wrinkle map.

[0081] In one possible implementation, the wrinkle image region is segmented to obtain multiple wrinkle image blocks, including: segmenting the wrinkle image region based on multiple segmentation windows to obtain initial image blocks of the wrinkle image region; determining the rotation parameters of each segmentation window based on each initial image block, wherein the rotation parameters of any segmentation window are the relevant parameters when rotating the segmentation window around a fixed point (which can be called the rotation center); rotating each segmentation window based on the rotation parameters of each segmentation window to obtain each rotated segmentation window, wherein rotating any segmentation window involves rotating the segmentation window around a fixed point (which can be called the rotation center); and segmenting the wrinkle image region based on each rotated segmentation window to obtain each wrinkle image block.

[0082] In this embodiment, the wrinkle image region is divided into blocks based on multiple block windows at different positions, so that each block window extracts an initial image block from the wrinkle image region, resulting in various initial image blocks. For any initial image block, the rotation parameters of the corresponding block window are determined based on the pixel information of each pixel in the initial image block. Based on the rotation parameters of the block window, the block window is rotated, and a wrinkle image block is extracted from the wrinkle image region based on the rotated block window.

[0083] Because the segmented window is rotated, after the wrinkle image region is divided into individual wrinkle image blocks based on the rotated segmented window, the direction of some edges of any wrinkle image block can point to the extension direction of the wrinkle. This makes the target wrinkle texture determined based on the wrinkle image block more expressive, thereby improving the expressiveness and realism of wrinkles in the face model with wrinkles.

[0084] Optionally, determining the rotation parameters of each block window based on each initial image block includes: for any initial image block, determining the gradient of each pixel in the horizontal direction and the gradient of each pixel in the vertical direction based on the pixel information of each pixel in the initial image block; and determining the rotation parameters of the block window corresponding to any initial image block based on the gradient of each pixel in the horizontal direction and the gradient of each pixel in the vertical direction.

[0085] Any initial image patch can be considered as consisting of multiple pixels arranged in multiple rows horizontally, and also as consisting of multiple pixels arranged in multiple columns vertically. Based on the pixel information of each pixel in any initial image patch, such as the three primary colors (Red-Green-Blue, RGB) data of each pixel in the initial image patch, the gradient of each pixel in the horizontal direction and the gradient of each pixel in the vertical direction can be calculated. Based on these two gradients, the rotation parameters of the corresponding block window of the initial image patch can be determined. Here, the horizontal direction can be denoted as the x-axis, the vertical direction can be denoted as the y-axis, and the rotation parameters include at least one of rotation angle and rotation direction.

[0086] Optionally, the pixel information (such as RGB data) of each pixel in any initial image block can be weighted and calculated to obtain the gradient of each pixel in the horizontal direction and the gradient of each pixel in the vertical direction in the initial image block.

[0087] In one possible implementation, the rotation parameters include a rotation angle. Based on the gradients of each pixel in the horizontal direction and the gradients of each pixel in the vertical direction of any initial image block, the rotation parameters of the block window corresponding to that initial image block are determined, including: determining the ratio between a first cumulative value and a second cumulative value, where the first cumulative value is the sum of the absolute values ​​of the gradients of each pixel in the horizontal direction and the second cumulative value is the sum of the absolute values ​​of the gradients of each pixel in the vertical direction; and determining the rotation angle of the block window corresponding to that initial image block based on this ratio. The rotation angle of the block window corresponding to any initial image block is the magnitude of rotation of the block window around a fixed point (which can be called the rotation center).

[0088] In this embodiment, the sum of absolute values ​​of the gradients of each pixel in the horizontal direction within any initial image block can be calculated, and this sum is denoted as the first cumulative value. Similarly, the sum of absolute values ​​of the gradients of each pixel in the vertical direction within the initial image block can be calculated, and this sum is denoted as the second cumulative value. Then, the ratio between the first and second cumulative values ​​is calculated, and the arctangent of this ratio is used as the rotation angle of the block window corresponding to the initial image block.

[0089] Optionally, the rotation angle of the block window corresponding to any initial image block can be determined according to the formula (1) shown below.

[0090] angle xy =arctan(∑ pixels abs(grad y ) / ∑ pixels abs(grad x )) Formula (1)

[0091] Among them, angle xy `r` represents the rotation angle of the block window corresponding to any initial image patch. `arctan` is the symbol for the arctangent function. `∑` is the symbol for the summation function. `pixels` is the total number of pixels in the initial image patch. `abs` is the symbol for the absolute value function. `grad` y This represents the gradient of any pixel in the initial image patch in the vertical direction (i.e., the y-axis direction). x Let be the gradient of the pixel in the initial image patch along the horizontal direction (i.e., the x-axis direction). Where ∑ pixels abs(grad y ) represents the second cumulative value mentioned above, ∑ pixels abs(grad x) represents the first cumulative value mentioned above.

[0092] The embodiments of this application determine the rotation parameters from the dimension of rotation angle, so that when the block window is rotated based on the rotation angle of the block window corresponding to any initial image block, the rotation amplitude of the block window can be accurately controlled, thereby improving the accuracy of wrinkle image blocks when dividing wrinkle image blocks from the wrinkle image region based on the block window.

[0093] In one possible implementation, the rotation parameters include the rotation direction; based on the gradients of each pixel in the horizontal direction and the gradients of each pixel in the vertical direction of any initial image block, the rotation parameters of the block window corresponding to any initial image block are determined, including: for any pixel in any initial image block, determining the maximum value corresponding to that pixel, where the maximum value is the maximum between the absolute value of the gradient of that pixel in the horizontal direction and the absolute value of the gradient of that pixel in the vertical direction; and based on the maximum values ​​corresponding to each pixel in any initial image block, determining the rotation direction of the block window corresponding to that initial image block. The rotation direction of the block window corresponding to any initial image block is the direction in which the block window is rotated around a fixed point (which can be called the rotation center).

[0094] For any pixel in any initial image patch, compare the absolute values ​​of its horizontal and vertical gradients. If the absolute value of the horizontal gradient is greater than the absolute value of its vertical gradient, then the horizontal gradient is taken as the maximum value for that pixel. If the absolute value of the horizontal gradient is less than the absolute value of its vertical gradient, then the vertical gradient is taken as the maximum value for that pixel. If the absolute value of the horizontal gradient is equal to the absolute value of its vertical gradient, then the absolute value of the gradient in either the horizontal (or vertical) direction is taken as the maximum value for that pixel.

[0095] Next, the product of the gradient of the pixel in the horizontal direction and the gradient of the pixel in the vertical direction can be calculated to obtain the first product value corresponding to the pixel. This first product value is then symbolized to obtain the symbolized first product value. In this embodiment, the rotation direction of the block window corresponding to any initial image block is determined based on the symbolized first product values ​​of each pixel in the initial image block and the maximum value of each pixel in the initial image block.

[0096] Optionally, for any pixel in any initial image block, the product of the symbolized first product value corresponding to that pixel and the maximum value corresponding to that pixel is calculated to obtain the second product value corresponding to that pixel. Then, the sum of the second product values ​​corresponding to each pixel in any initial image block is calculated, and the obtained sum is symbolized to obtain the rotation direction of the block window corresponding to that initial image block.

[0097] Optionally, the rotation direction of the block window corresponding to any initial image block can be determined according to the formula (2) shown below.

[0098] dir xy =sign(∑ pixels sign(grad y *grad x )*max(abs(grad y ),abs(grad x ))) Formula (2)

[0099] Where, dir xy This represents the rotation direction of the block window corresponding to any initial image patch. `sign` is the function symbol for the sign function, used to symbolize the independent variable. `∑` is the function symbol for the summation function. `pixels` is the total number of pixels in the initial image patch. `grad` y This represents the gradient of any pixel in the initial image patch in the vertical direction (i.e., the y-axis direction). x This represents the gradient of the pixel in the initial image patch along the horizontal direction (x-axis). `max` is the function notation for the maximum value. `abs` is the function notation for the absolute value. Where `max(abs(grad)`... y ),abs(grad x The sign(grad) represents the maximum value corresponding to the pixel in the initial image patch mentioned above. y *grad x The first product value after symbolization of the pixel in the initial image block mentioned above is represented, where the maximum value can be used as the weight of the first product value.

[0100] The embodiments of this application determine the rotation parameters from the dimension of the rotation direction, so that when the block window is rotated based on the rotation direction of the block window corresponding to any initial image block, the rotation direction of the block window can be accurately controlled, thereby improving the accuracy of wrinkle image blocks when dividing wrinkle image blocks from the wrinkle image region based on the block window.

[0101] This allows us to determine the rotation parameters (including at least one of rotation angle and rotation direction) of the segmented window corresponding to any initial image patch. Next, based on these rotation parameters, the segmented window is rotated to obtain a rotated segmented window. Then, based on this rotated window, wrinkle image patches are extracted from the wrinkle image region, with each initial image patch corresponding to one wrinkle image patch. This method achieves segmentation of the wrinkle image region, resulting in multiple wrinkle image patches. Because the segmented window is rotated, the directions of some edges of the wrinkle image patches extracted from the wrinkle image region can point towards the wrinkle's extension direction, making the target wrinkle texture determined based on the wrinkle image patches more expressive, thereby improving the expressiveness and realism of wrinkles in the facial model with wrinkles.

[0102] Please see Figure 3 , Figure 3 This is a schematic diagram illustrating a process for determining wrinkle image blocks according to an embodiment of this application. In this embodiment, wrinkle image regions can be determined by filtering a reference facial image. Specifically, filtering the reference facial image yields a filtering result, which includes the probability that each pixel in the reference facial image belongs to a pixel corresponding to a wrinkle. Based on the filtering result, wrinkle image regions in the reference facial image can be determined. Dividing the wrinkle image regions into blocks yields multiple wrinkle image blocks, wherein... Figure 3 The diagram shows how segmenting a portion of the wrinkle image region (i.e., the region within the dashed box) results in multiple wrinkle image blocks.

[0103] Step 203: Determine the target wrinkle map based on multiple wrinkle image blocks.

[0104] In this embodiment, the target wrinkle map is a normal map used to represent wrinkles in a reference facial image. A normal map is created by plotting normals at each point on the uneven surface of the original object, and the direction of the normals is marked using color channels. A normal map can be understood as another different surface parallel to the original uneven surface, but in reality, it is a smooth plane. The target wrinkle map includes at least one wrinkle region, and any wrinkle region is the mapping area where the wrinkle is located in the target wrinkle map.

[0105] In one possible implementation, determining a target wrinkle map based on multiple wrinkle image patches includes: for any wrinkle image patch, determining the features of the corresponding texture image patch based on the wrinkle image patch and a feature processing model; determining a target displacement map based on the features of the corresponding texture image patches of each wrinkle image patch, wherein the target displacement map is a displacement map used to represent wrinkles in a reference facial image; and converting the target displacement map into a target wrinkle map.

[0106] A neural network model can be trained to obtain a feature processing model. This model controls the feature processing model to determine the features of a corresponding texture image patch based on any given wrinkle image patch. This texture image patch is an image patch in a displacement map. The value of any pixel in the displacement map represents the distance between that pixel and the plane defined by the displacement map, thereby achieving the effect of moving the pixel's position a corresponding distance along the normal of the plane. Optionally, the feature processing model can be a regressor model.

[0107] Optionally, before determining the features of the texture image block corresponding to any wrinkle image block based on any wrinkle image block and the feature processing model, the method further includes: acquiring a sample face image and a sample permutation map corresponding to the sample face image, wherein the sample face image includes the sample image region where the wrinkle is located, and the sample permutation map includes the sample texture region where the wrinkle is located; performing block processing on the sample image region to obtain each first image block; performing block processing on the sample texture region to obtain each second image block; and training the neural network model based on each first image block and each second image block to obtain the feature processing model.

[0108] In this embodiment, a sample facial image can be acquired. The method for acquiring the sample facial image is similar to that for acquiring the reference facial image, and will not be repeated here. Based on a principle similar to that of step 202, the image region where wrinkles are located in the sample facial image can be determined. This image region is called the sample image region. The sample image region is then divided into blocks to obtain each first image block. The specific implementation method can be found in the relevant description of step 202, and will not be repeated here.

[0109] In this embodiment, a sample displacement map corresponding to the sample facial image can also be obtained. The sample displacement map is a displacement map used to represent wrinkles in the sample facial image. Optionally, a sample wrinkle map corresponding to the sample facial image can be obtained first. The sample wrinkle map is a normal map used to represent wrinkles in the sample facial image. After obtaining the sample wrinkle map, it can be transformed to obtain the sample displacement map. Based on a principle similar to that of step 202, the mapping region where the wrinkles are located in the sample displacement map can be determined. This mapping region is called the sample mapping region, and the sample mapping region is divided into blocks to obtain various second image blocks. Specific implementation details can be found in the relevant description of step 202, and will not be repeated here.

[0110] Please see Figure 4 , Figure 4 This is a schematic diagram illustrating the acquisition of an image block set according to an embodiment of this application. The image block set in this embodiment includes multiple first image blocks and multiple second image blocks. Each first image block is obtained by segmenting a sample image region. Each second image block is obtained by converting a sample wrinkle map into a sample displacement map, and then segmenting the sample map region containing the wrinkles in the sample displacement map.

[0111] It should be noted that the sample facial images and sample displacement maps can be obtained based on facial processing software (such as MetaHuman, a facial processing software). Facial processing software includes multiple facial models, multiple displacement maps, multiple normal maps, etc. Using facial processing software, different facial expressions can be set, and a sample facial image of a target size (e.g., 1024*1024) can be rendered. The sample facial image can then be divided into blocks (e.g., through screenshotting) to obtain various first image blocks. Using facial processing software, the sample displacement map corresponding to the sample facial image is obtained, and its size is set to the target size. The sample displacement map is then divided into blocks (e.g., through screenshotting) to obtain second image blocks corresponding to each first image block. A first image block and a second image block are considered as a pair of image data. This application embodiment does not limit the number of image data pairs. For example, the number of image data pairs is 6651 pairs, meaning there are 6651 first image blocks and 6651 second image blocks.

[0112] After obtaining multiple first image blocks and multiple second image blocks in the above manner, the neural network model is trained based on each first image block and each second image block to obtain the feature processing model.

[0113] In one possible implementation, a first image patch corresponds to a second image patch; based on each first image patch and each second image patch, a neural network model is trained to obtain a feature processing model, including: acquiring second initial features, which are input information set for the neural network model; for any first image patch, controlling the neural network model to modify the second initial features according to the first image patch to obtain the first feature of the second image patch corresponding to the first image patch; extracting the second features of each second image patch; and adjusting the neural network model based on the first and second features of each second image patch to obtain the feature processing model.

[0114] The number of first image blocks and second image blocks is the same, and there is a one-to-one correspondence between the first image blocks and the second image blocks. Optionally, when dividing the sample image region into blocks to obtain each first image block, and dividing the sample texture region into blocks to obtain each second image block, a principle similar to "dividing the wrinkle image region into blocks to obtain multiple wrinkle image blocks" can be used. First, the sample image region is divided into blocks based on multiple block windows to obtain each initial sample block. Then, the rotation parameters of each block window are determined based on each initial sample block. Next, each block window is rotated based on its rotation parameters to obtain each rotated block window. Then, the sample image region is divided into blocks based on each rotated block window to obtain each first image block, and the sample texture region is divided into blocks based on each rotated block window to obtain each second image block. The first image blocks and second image blocks obtained based on the same rotated block window have a corresponding relationship.

[0115] In this embodiment, a second initial feature can be obtained. Optionally, feature extraction is performed on the sample texture region to obtain the features of the sample texture region. The features of the sample texture region are divided by the number of second image blocks to obtain the average value of the first feature, and the average value of the first feature is used as the second initial feature. Alternatively, feature extraction can be performed on each second image block to obtain the second features of each second image block. The average value of the second feature is determined based on the second features of each second image block, and the average value of the second feature is used as the second initial feature.

[0116] This application does not limit the method of feature extraction. For example, taking a sample texture region (similar in principle to the second image patch) as an example, Principal Component Analysis (PCA) can be used to extract features from the sample texture region, resulting in PCA features. Alternatively, the Histogram of Oriented Gradient (HOG) algorithm can be used to extract features from the sample texture region, resulting in HOG features. Furthermore, a color histogram algorithm can be used to extract features from the sample texture region, resulting in color histogram features.

[0117] In this embodiment, the second initial feature can be input into the neural network model, and the neural network model can be controlled to modify the second initial feature according to any first image block to obtain the first feature of the second image block corresponding to the first image block.

[0118] Since the first feature of the second image patch is obtained by modifying the second initial feature, the first feature and the second initial feature of the second image patch have the same feature type. That is, if the second initial feature is a PCA feature (i.e., the second initial feature is determined based on the PCA features of the sample texture region or each of the second image patches), then the first feature of the second image patch is also a PCA feature. Similarly, if the second initial feature is a HOG feature (i.e., the second initial feature is determined based on the HOG features of the sample texture region or each of the second image patches), then the first feature of the second image patch is also a HOG feature. If the second initial feature is a color histogram feature (i.e., the second initial feature is determined based on the color histogram features of the sample texture region or each of the second image patches), then the first feature of the second image patch is also a color histogram feature.

[0119] Optionally, the neural network model includes at least one cascaded initial classifier; controlling the neural network model to modify the second initial feature according to any first image patch to obtain the first feature of the second image patch corresponding to any first image patch includes: inputting the second initial feature into at least one cascaded initial classifier to obtain the output information of each initial classifier, wherein the output information of the previous initial classifier in the at least one cascaded initial classifier is the input information of the next initial classifier, and the output information of the last initial classifier is the first feature of the second image patch corresponding to any first image patch.

[0120] Please see Figure 5 , Figure 5This is a schematic diagram illustrating the generation of a first feature of a second image according to an embodiment of this application. By inputting the second initial feature into at least one cascaded initial classifier, the first feature of the second image block can be obtained. Specifically, the second initial feature serves as the input information for the first initial classifier, and the output information for the last initial classifier serves as the first feature of the second image block.

[0121] Optionally, for any initial classifier, the output information of the initial classifier is determined using a semi-naive Bayes approach. The initial classifier can be controlled to correct its input information based on any first image patch to obtain its output information. In this case, the output information of the last initial classifier is used as the first feature of the second image patch corresponding to that first image patch.

[0122] For any initial classifier, its structure is similar to a multi-level binary classifier. This initial classifier can be controlled to perform the following operations: its input information is used as the input information of the first-level binary classifier in the multi-level binary classifier; the output information of the last-level binary classifier in the multi-level binary classifier is used as the output information of the initial classifier; and the output information of the previous-level binary classifier in the multi-level binary classifier is used as the input information of the next-level binary classifier. For any level binary classifier in the multi-level binary classifier, it can generate a random number and compare the random number with the threshold of the initial classifier. If the random number is greater than the threshold, the input information of the binary classifier is classified as the first category, and the input information is corrected based on the feature correction algorithm corresponding to the first category to obtain the output information of the binary classifier. If the random number is not greater than the threshold, the input information of the binary classifier is classified as the second category, and the input information is corrected based on the feature correction algorithm corresponding to the second category to obtain the output information of the binary classifier.

[0123] Please see Figure 5 The initial classifier shown uses a path from the previous black circle to the next black circle to represent the input information (corresponding to the previous black circle) of the binary classifier, which is modified based on the feature correction algorithm corresponding to the first or second category, to obtain the output information (corresponding to the next black circle). The downward-sloping arrow to the left corresponds to the first category, and the downward-sloping arrow to the right corresponds to the second category. Figure 5 It can be seen that the structure of the initial classifier is similar to that of a multi-level binary classifier.

[0124] The threshold of the initial classifier is determined based on any first image patch as follows: First, a first number of pixels are randomly selected from any first image patch, denoted as K, where K is a positive integer. Then, using the pixel information of each of the K pixels, the difference between every two pixels in the K pixels is calculated to obtain a second number of differences, denoted as K. 2 Next, from K 2 Select a third number of differences that are greater than the reference difference, and denote this third number as F, where F is a positive integer. Use these F differences as the threshold for the initial classifier, or calculate the threshold for the initial classifier based on these F differences. Optionally, the number of differences corresponding to the previous initial classifier is less than or equal to the number of differences corresponding to the next initial classifier (i.e., F corresponding to the previous initial classifier is less than or equal to F corresponding to the next initial classifier).

[0125] As mentioned above, the structure of any initial classifier is similar to that of a multi-stage binary classifier, where the number of stages of the binary classifiers corresponding to any two initial classifiers can be the same or different. Optionally, in at least one cascaded initial classifier, the number of stages of the binary classifier corresponding to the previous initial classifier is less than or equal to the number of stages of the binary classifier corresponding to the next initial classifier.

[0126] In this embodiment of the application, the relationship between the input information and the output information of any initial classifier is shown in the following formula (3).

[0127]

[0128] Where δs represents the output information of the initial classifier, ∑ is the function sign of the summation function, σ represents the number of second image patches, and s gt The second feature representing the i-th second image patch, s i Let |σ| represent the first feature of the i-th second image patch. Let |σ| represent the distribution of the number of second image patches corresponding to each correction path when the first features of the σ second image patches are corrected according to each correction path of the initial classifier. Each correction path includes a feature correction algorithm corresponding to the first or second class of each level of the multi-level binary classifier. The number of correction paths is 2. N N is a positive integer, and N represents the number of stages in the multi-stage binary classifier. Optionally, N can be the third quantity mentioned above.

[0129] For example, for a two-stage binary classifier, one correction approach could include feature correction algorithms for the first class of the first-stage binary classifier and for the first class of the second-stage binary classifier. Another correction approach could include feature correction algorithms for the second class of the first-stage binary classifier and for the first class of the second-stage binary classifier.

[0130] It should be noted that in formula (3), when the number of second image patches is less than the reference number, a shrinkage parameter can be set for |σ|. This shrinkage parameter is used to characterize the attenuation information of the number of second image patches corresponding to each correction path. When the number of second image patches is not less than the reference number, no shrinkage parameter is set for |σ|. In this way, when the number of second image patches is small, the fitting ability of the initial classifier can be improved, and the wrinkle representation ability can be enhanced. When the number of second image patches is large, the overfitting of the initial classifier can be prevented, and the wrinkle representation ability can be enhanced.

[0131] In this embodiment, second features of each second image block can also be extracted. The method for extracting the second feature of any second image block is similar to the feature extraction method mentioned above, which involves "extracting features from the sample texture region to obtain the features of the sample texture region," and will not be repeated here. Furthermore, the first feature of the second image block and the second feature of the second image block have the same feature type.

[0132] Then, based on the first features of each second image block and the second features of each second image block, the loss value of the neural network model is determined according to the formula (4) shown below.

[0133]

[0134] Equation (4) can be considered as an energy function. reg ∑ represents the loss value of the neural network model, ∑ represents the function sign of the summation function, and N represents the number of the second image patches. c is the first feature of the i-th second image patch. i Let A be the second feature of the i-th second image patch. ||A|| represents the norm of A, where A is the independent variable.

[0135] The neural network model is adjusted based on its loss value to obtain an adjusted neural network model. If the training termination condition is met, such as reaching the target number of training iterations (e.g., 500 times) or the gradient of the neural network model's loss value no longer decreasing, the adjusted neural network model is used as the feature processing model. If the training termination condition is not met, the adjusted neural network model is used as the neural network model for the next training iteration. By controlling the neural network model, the first features of each second image patch are obtained. Based on the first features of each second image patch and the second image, the neural network model is adjusted again until the training termination condition is met, resulting in the feature processing model.

[0136] By continuously adjusting the neural network model, the first feature of the second image block can be made to continuously approximate the second feature of the second image block, thereby reducing the error between the first feature and the second feature of the second image block and improving the accuracy of the first feature.

[0137] In one possible implementation, a feature processing model is obtained by adjusting the neural network model based on the first and second features of each second image patch, including: for any one of the at least one cascaded initial classifiers, determining the loss value of any one initial classifier based on the input information of any one initial classifier, the second features of each second image patch, and the output information of any one initial classifier; and adjusting the neural network model based on the loss values ​​of each of the at least one cascaded initial classifiers to obtain the feature processing model.

[0138] The loss value of the initial classifier can be determined based on the input information of any initial classifier, the second features of each second image patch, and the output information of the initial classifier, according to the formula (5) shown below. The loss value of the neural network model is then determined based on the loss values ​​of each initial classifier. Each second image patch corresponds to one input information of the initial classifier and one output information of the initial classifier.

[0139]

[0140] Where E represents the loss value of the neural network model, ||s gt -s i-1 -Reg(i t )|| represents the loss value of the i-th initial classifier. argmin represents the loss value of the function. Minimum. ∑ is the function sign of the summation function, and N is the initial number of classifiers. s gt The second feature representing each second image patch, s i-1 Represents the input information of the i-th initial classifier corresponding to each second image patch, Reg(i t) represents the output information of the i-th initial classifier corresponding to each second image patch. ||A|| represents the norm of A, where A is the independent variable.

[0141] Optionally, the loss value of any initial classifier can be used to adjust the initial classifier. By continuously adjusting the initial classifier, the neural network model can be continuously adjusted based on formula (5), so that the first feature of the second image patch can continuously approach the second feature of the second image patch, thereby reducing the error between the first feature of the second image patch and the second feature of the second image patch and improving the accuracy of the first feature.

[0142] After training the feature processing model, it can be controlled to modify the first initial feature based on any wrinkle image patch, thereby obtaining the feature of the corresponding texture image patch. The first initial feature and the second initial feature are the same.

[0143] Optionally, the feature processing model includes at least one cascaded random fern classifier; based on any wrinkle image patch and the feature processing model, determining the features of the texture image patch corresponding to any wrinkle image patch includes: obtaining a first initial feature, the first initial feature being input information set for the feature processing model; inputting the first initial feature into at least one cascaded random fern classifier, and controlling at least one cascaded random fern classifier to perform multiple corrections on the first initial feature according to any wrinkle image patch to obtain the features of the texture image patch corresponding to any wrinkle image patch.

[0144] Understandably, the random fern classifier is obtained by adjusting the initial classifier. Therefore, the number of random fern classifiers is the same as the number of initial classifiers, and the structure of the random fern classifier is also the same as the structure of the initial classifier; the only difference is the model parameters. In other words, the structure of the random fern classifier is similar to that of a multi-level binary classifier.

[0145] Optionally, for any random fern classifier, the classifier can be controlled to modify its input information based on any wrinkle image patch to obtain its output information. In this case, the first initial feature serves as the input information of the first random fern classifier, and the output information of the last random fern classifier serves as the feature of the texture image patch corresponding to the wrinkle image patch.

[0146] For any given random fern classifier, its structure is similar to a multi-level binary classifier. The random fern classifier can be controlled to perform the following operations: its input information is used as the input information of the first-level binary classifier in the multi-level binary classifier; the output information of the last-level binary classifier in the multi-level binary classifier is used as the output information of the random fern classifier; and the output information of the previous-level binary classifier in the multi-level binary classifier is used as the input information of the next-level binary classifier. For any level binary classifier in the multi-level binary classifier, the classifier can generate a random number and compare the random number with the threshold of the random fern classifier. If the random number is greater than the threshold, the input information of the random fern classifier is classified as the first category, and the input information is corrected based on the feature correction algorithm corresponding to the first category to obtain the output information of the binary classifier. If the random number is not greater than the threshold, the input information of the random fern classifier is classified as the second category, and the input information is corrected based on the feature correction algorithm corresponding to the second category to obtain the output information of the binary classifier.

[0147] The threshold of this random fern classifier is determined based on any wrinkle image patch as follows: First, a first number of pixels are randomly selected from any wrinkle image patch, denoted as K, where K is a positive integer. Then, using the pixel information of each of the K pixels, the difference between every two pixels in the K pixels is calculated to obtain a second number of differences, denoted as K. 2 Next, from K 2 Select a third number of differences that are greater than the reference difference from the initial difference, and denote this third number as F, where F is a positive integer. Use these F differences as the threshold for the random fern classifier, or calculate the threshold for the random fern classifier based on these F differences.

[0148] It should be noted that the implementation method of "obtaining the features of the texture image block corresponding to any wrinkle image block" can be found in the implementation content above regarding the determination of "the first feature of the second image block corresponding to any first image block". Here, the first initial feature corresponds to the second initial feature mentioned above, the random fern classifier corresponds to the initial classifier mentioned above, any wrinkle image block corresponds to the first image block mentioned above, the texture image block corresponds to the second image block mentioned above, and the feature of the texture image block corresponds to the first feature of the second image block mentioned above.

[0149] The features of the texture image block corresponding to any wrinkle image block can be obtained through the above method. Optionally, the features of the texture image block corresponding to the wrinkle image block can be inversely transformed (e.g., upsampling) to obtain the texture image block corresponding to the wrinkle image block. The texture image blocks corresponding to each wrinkle image block are then stitched together to obtain the stitched region. Since the texture image block is an image block in the displacement map, the target displacement map can be determined based on the stitched region. Optionally, the features of the texture image blocks corresponding to each wrinkle image block are stitched together to obtain stitched features. The stitched features are then inversely transformed to obtain the stitched region, and the target displacement map is determined based on the stitched region.

[0150] When determining the target displacement map based on the stitched region, the stitched region can first undergo edge processing to obtain the edge processing result, and then the target displacement map can be determined based on the edge processing result. Optionally, the stitched region can be dilated to obtain a dilated stitched region, and then compressed to obtain a compressed stitched region. The difference between the dilated and compressed stitched regions is determined, and Poisson editing is performed on this difference to obtain the edge processing value of each pixel in the stitched region, thus obtaining the edge processing result.

[0151] After obtaining the target displacement map, image processing software (such as Photoshop) can be used to convert it into a target wrinkle map. This target wrinkle map includes at least one wrinkle region, which is the map area where wrinkles are located. Optionally, a masking process can be applied to the target wrinkle map to extract the wrinkle regions.

[0152] Step 204: Determine the model region in the face model to be processed. The model region is the area on the face model to be processed that corresponds to the target wrinkle texture.

[0153] The facial model to be processed includes multiple vertices. Wrinkles are typically created by squeezing the skin; therefore, a blended deformation model can be obtained by performing a blend deformation on the facial model. This blended deformation model can then reproduce the expressions (such as smiling, crying, pouting, etc.) from the reference facial image. By determining the distance between every two adjacent vertices in the blended deformation model, the activated regions are identified. These activated regions represent the wrinkles within the blended deformation model; therefore, there is a correspondence between the activated regions and the target wrinkle texture.

[0154] Optionally, for any vertex in the hybrid deformed model, the distance between the vertex and its adjacent vertices is determined using the vertex's position information within the hybrid deformed model and the position information of its neighboring vertices (i.e., vertices adjacent to the vertex). If the distance between the vertex and its neighboring vertices is less than a distance threshold, the vertex is marked as a hybrid deformed vertex. The region in the hybrid deformed model that contains each hybrid deformed vertices is called the activated region.

[0155] The blended deformed model is obtained by blending deformers onto the facial model to be processed. Therefore, every vertex in the blended deformed model can be found in the facial model to be processed. The activated region is the area of ​​wrinkles in the blended deformed model, and the activated region includes multiple vertices. The corresponding vertices of each vertex in the activated region can be found in the facial model to be processed. The area in the facial model to be processed containing the corresponding vertices is called the model region. There is a correspondence between the model region and the activated region. Since there is a correspondence between the activated region and the target wrinkle texture, there is also a correspondence between the model region and the target wrinkle texture.

[0156] Step 205: Integrate the target wrinkle texture into the model area to obtain a facial model with wrinkles.

[0157] In this embodiment, the activated region can be stored. Optionally, the UV coordinates of each vertex in the activated region can be stored to obtain a vertex motion map, wherein the UV coordinates of the vertex can represent the position of the vertex in a two-dimensional texture coordinate system, where the horizontal coordinate of the two-dimensional texture coordinate system is the U coordinate and the vertical coordinate is the V coordinate.

[0158] It should be noted that a vertex motion map is a single-channel map. Therefore, three vertex motion maps can be stored in the manner of storing three-channel images (including red, green, and blue channels) to improve storage efficiency.

[0159] To save storage space, if at least two vertex motion maps contain overlapping regions, the overlapping regions are stored only once. For at least two vertex motion maps containing fewer than a certain number of vertices, regardless of whether the two vertex motion maps overlap, only one of the vertex motion maps needs to be stored.

[0160] Since vertex motion maps are obtained by storing the UV coordinates of each vertex in the activated region, there is a correspondence between vertex motion maps and the activated region. Since there is a correspondence between the model region and the activated region, there is also a correspondence between vertex motion maps and the model region.

[0161] In this embodiment of the application, when the target wrinkle map is fused onto the model area, the model area in the face model to be processed can be fused and deformed according to the fusion weight, vertex movement map and target wrinkle map to obtain a face model with wrinkles.

[0162] Please see Figure 6 , Figure 6 This is a schematic diagram illustrating the deformation of a facial model according to an embodiment of this application. By performing a blended deformation on the facial model to be processed, a vertex movement map can be obtained. Based on the vertex movement map and the target wrinkle map, the model regions in the facial model to be processed are blended and deformed to obtain a facial model with wrinkles.

[0163] It should be noted that since the target wrinkle map includes at least one wrinkle region, the model region can be divided based on this at least one wrinkle region, resulting in sub-regions corresponding to each wrinkle region within the model region. When fusing the target wrinkle map onto the model region, each wrinkle region can be fused to its corresponding sub-region separately, achieving fine-grained fusion and improving the expressiveness of wrinkles in local areas of a wrinkled facial model.

[0164] It is understood that the embodiments of this application are not limited to using steps 204, 205, and other related content to fuse the target wrinkle map onto the model area. Optionally, a method of segmenting the facial model or using dense facial model feature points can also be used to fuse the target wrinkle map onto the model area, which will not be elaborated here.

[0165] The facial model processing method provided in this application can be applied to various scenarios requiring the generation of facial models with wrinkles. For example, when acquiring a user-created facial expression image, which is a facial image with wrinkles, the facial model processing method provided in this application can generate a facial model with wrinkles that matches the expression in the image, improving the quality of user-created content. Alternatively, in game scenarios, it is often necessary to process the faces of a large number of game models. The facial model processing method provided in this application can quickly generate facial models with wrinkles, saving model designers the time required to design game models.

[0166] Optionally, the processing shown in steps 201-205 can be implemented using a plugin. This plugin can be coded in Python (a computer programming language), while the feature processing model can be coded in an object-oriented programming language (such as C++). Optionally, for the C++ file coded in C++, a corresponding Python .pyd file can be generated using Pybind11 (a lightweight C++ library), allowing direct calling of the .pyd file from Python, thereby enabling direct invocation of the feature processing model. Through this plugin, a wrinkled facial model corresponding to a reference facial image can be directly obtained from that reference facial image.

[0167] It should be noted that all information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals involved in this application are authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the images involved in this application, such as reference facial images, sample facial images, and sample replacement maps, were all obtained with full authorization.

[0168] The above method divides the wrinkle image region of the reference facial image into multiple wrinkle image blocks, and determines the target wrinkle texture based on these multiple wrinkle image blocks. This reduces the time required to acquire the wrinkle texture and improves the efficiency of wrinkle texture acquisition. Since the target wrinkle texture is fused to the model region corresponding to the target wrinkle texture, a wrinkled facial model is obtained. Therefore, the time required to acquire a wrinkled facial model is reduced, the efficiency of acquiring a wrinkled facial model is improved, and the processing efficiency of the facial model is also improved.

[0169] The above describes the facial model processing method from the perspective of methodology and steps. Below, we will combine... Figure 7 The following describes in detail the facial model processing method of the embodiments of this application. Figure 7 This is a schematic diagram of a process for determining a wrinkle region provided in an embodiment of this application.

[0170] In this embodiment, the original image is first acquired, and a face fitting is performed on the original image to obtain a reference face image. Optionally, the EOS face model fitting library is used to perform face fitting on the original image to fit the face of the organism in the original image, thereby obtaining the reference face image. Then, the reference face image is divided into blocks to obtain multiple wrinkle image blocks. The block division method can be found in the relevant description of step 202, and will not be repeated here. Next, feature correction processing is performed based on each wrinkle image block to obtain multiple texture image blocks.

[0171] In the process of obtaining a texture image patch by performing feature correction processing based on any wrinkle image patch, a feature processing model can be trained first. This model is then used to modify the initial features multiple times based on any wrinkle image patch to obtain the features of the texture image patch corresponding to that wrinkle image patch. Next, the features of the texture image patch corresponding to that wrinkle image patch are subjected to inverse transformation processing to obtain the texture image patch itself.

[0172] Next, the target displacement map is obtained by stitching together multiple texture image blocks. The target displacement map is then converted into a target wrinkle map. Finally, a mask is applied to the target wrinkle map to obtain the wrinkle regions within it.

[0173] It should be noted that step 203 above describes determining the target wrinkle map based on multiple wrinkle image patches, which will not be repeated here.

[0174] In this embodiment, the model region corresponding to the target wrinkle texture can be determined in the facial model to be processed. The method for determining the model region is described in step 204 and will not be repeated here. Next, the target wrinkle texture is fused to the model region to obtain a facial model with wrinkles. The method for fusion of the target wrinkle texture to the model region is described in step 205 and will not be repeated here.

[0175] Please see Figure 8 , Figure 8 This is a schematic diagram of an original image and a corresponding facial model with wrinkles provided in an embodiment of this application. Wherein, Figure 8 1(a) in the image is an original image. Figure 8 In the image, 1(b) is the face model with wrinkles corresponding to the original image 1(a). Figure 8 2(a) in the image is another original image. Figure 8 Image 2(b) in the diagram is the wrinkled facial model corresponding to the original image 2(a). (The rest of the text appears to be a fragment and requires further context for accurate translation.) Figure 8 As can be seen, the facial model with wrinkles can present the deeper and more obvious wrinkles in the original image without presenting noise such as pores in the original image. Therefore, the facial model with wrinkles has a better wrinkle representation capability.

[0176] Figure 9 The diagram shown is a structural schematic of a facial model processing device provided in an embodiment of this application. Figure 9 As shown, the device includes:

[0177] The acquisition module 901 is used to acquire a reference facial image, which includes a wrinkle image region;

[0178] The segmentation module 902 is used to segment the wrinkle image region into multiple wrinkle image blocks.

[0179] The determination module 903 is used to determine the target wrinkle map based on multiple wrinkle image blocks;

[0180] The determination module 903 is used to determine the model region in the face model to be processed. The model region is the region on the face model to be processed that corresponds to the target wrinkle texture.

[0181] The fusion module 904 is used to fuse the target wrinkle map onto the model area to obtain a facial model with wrinkles.

[0182] In one possible implementation, the device further includes:

[0183] The filtering module is used to filter the reference facial image to obtain the filtering result;

[0184] The determination module 903 is also used to determine the probability that each pixel in the reference facial image belongs to the pixel corresponding to the wrinkle based on the filtering result; and to determine the wrinkle image region based on the probability that each pixel in the reference facial image belongs to the pixel corresponding to the wrinkle.

[0185] In one possible implementation, a filtering module is used to perform differential filtering on the reference facial image to obtain a differential filtering result; and to perform average filtering on the differential filtering result to obtain an average filtering result.

[0186] In one possible implementation, the segmentation module 902 is used to segment the wrinkle image region based on multiple segmentation windows to obtain each initial image block; determine the rotation parameters of each segmentation window based on each initial image block; rotate each segmentation window based on the rotation parameters of each segmentation window to obtain each rotated segmentation window; and segment the wrinkle image region based on each rotated segmentation window to obtain each wrinkle image block.

[0187] In one possible implementation, the segmentation module 902 is used to, for any initial image block, determine the gradient of each pixel in the horizontal direction and the gradient of each pixel in the vertical direction based on the pixel information of each pixel in the initial image block; and determine the rotation parameters of the segmented window corresponding to the initial image block based on the gradient of each pixel in the horizontal direction and the gradient of each pixel in the vertical direction.

[0188] In one possible implementation, the rotation parameters include the rotation angle;

[0189] The block segmentation module 902 is used to determine the ratio between the first cumulative value and the second cumulative value. The first cumulative value is the sum of the absolute values ​​of the gradients of each pixel in any initial image block in the horizontal direction, and the second cumulative value is the sum of the absolute values ​​of the gradients of each pixel in any initial image block in the vertical direction. Based on the ratio, the rotation angle of the block window corresponding to any initial image block is determined.

[0190] In one possible implementation, the rotation parameters include the rotation direction;

[0191] The block segmentation module 902 is used to determine the maximum value corresponding to any pixel in any initial image block. The maximum value corresponding to any pixel is the maximum value between the absolute value of the gradient of any pixel in the horizontal direction and the absolute value of the gradient of any pixel in the vertical direction. Based on the maximum values ​​corresponding to each pixel in any initial image block, the rotation direction of the block window corresponding to any initial image block is determined.

[0192] In one possible implementation, the determining module 903 is used to determine the features of the texture image block corresponding to any wrinkle image block based on the wrinkle image block and the feature processing model; determine the target displacement map based on the features of the texture image blocks corresponding to each wrinkle image block; and convert the target displacement map into a target wrinkle map.

[0193] In one possible implementation, the feature processing model includes at least one cascaded random fern classifier;

[0194] The determination module 903 is used to obtain a first initial feature, which is the input information set for the feature processing model; the first initial feature is input into at least one cascaded random fern classifier, and the at least one cascaded random fern classifier is controlled to perform multiple corrections on the first initial feature based on any wrinkle image patch to obtain the feature of the texture image patch corresponding to any wrinkle image patch.

[0195] In one possible implementation, the acquisition module 901 is further configured to acquire a sample face image and a sample permutation map corresponding to the sample face image, wherein the sample face image includes the sample image region where the wrinkles are located, and the sample permutation map includes the sample map region where the wrinkles are located.

[0196] The block segmentation module 902 is also used to segment the sample image region into blocks to obtain each first image block; and to segment the sample texture region into blocks to obtain each second image block.

[0197] The acquisition module 901 is also used to train the neural network model based on each first image block and each second image block to obtain a feature processing model.

[0198] In one possible implementation, one first image block corresponds to one second image block;

[0199] The acquisition module 901 is used to acquire the second initial feature, which is the input information set for the neural network model; for any first image block, the neural network model is controlled to modify the second initial feature according to the first image block to obtain the first feature of the second image block corresponding to the first image block; the second feature of each second image block is extracted; based on the first feature and the second feature of each second image block, the neural network model is adjusted to obtain the feature processing model.

[0200] In one possible implementation, the neural network model includes at least one cascaded initial classifier;

[0201] The acquisition module 901 is used to input the second initial feature into at least one cascaded initial classifier to obtain the output information of each initial classifier. The output information of the previous initial classifier in the at least one cascaded initial classifier is the input information of the next initial classifier, and the output information of the last initial classifier is the first feature of the second image block corresponding to any first image block.

[0202] The acquisition module 901 is used to determine the loss value of any initial classifier in at least one cascaded initial classifier based on the input information of any initial classifier, the second features of each second image patch, and the output information of any initial classifier; and to adjust the neural network model based on the loss values ​​of each initial classifier in at least one cascaded initial classifier to obtain a feature processing model.

[0203] The aforementioned device divides the wrinkle image region of a reference facial image into multiple wrinkle image blocks, and determines the target wrinkle texture based on these multiple wrinkle image blocks. This reduces the time required to acquire the wrinkle texture and improves the efficiency of wrinkle texture acquisition. Since the target wrinkle texture is fused to the model region corresponding to the target wrinkle texture to obtain a wrinkled facial model, the time required to acquire the wrinkled facial model is reduced, the efficiency of acquiring the wrinkled facial model is improved, and the processing efficiency of the facial model is also increased.

[0204] It should be understood that the above Figure 9 The provided device, in implementing its functions, is only illustrated by the division of the above-described functional modules. In practical applications, the functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. Furthermore, the device and method embodiments provided in the above embodiments belong to the same concept, and their specific implementation processes are detailed in the method embodiments, and will not be repeated here.

[0205] Figure 10 This diagram illustrates a structural block diagram of a terminal device 1000 provided in an exemplary embodiment of this application. The terminal device 1000 includes a processor 1001 and a memory 1002.

[0206] Processor 1001 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 1001 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). Processor 1001 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 1001 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content required to be displayed on the screen. In some embodiments, processor 1001 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0207] The memory 1002 may include one or more computer-readable storage media, which may be non-transitory. The memory 1002 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 1002 are used to store at least one computer program, which is executed by the processor 1001 to implement the facial model processing method provided in the method embodiments of this application.

[0208] In some embodiments, the terminal device 1000 may also optionally include: a peripheral device interface 1003 and at least one peripheral device. The processor 1001, memory 1002, and peripheral device interface 1003 can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface 1003 via a bus, signal line, or circuit board. Specifically, the peripheral device includes at least one of: a radio frequency circuit 1004, a display screen 1005, a camera assembly 1006, an audio circuit 1007, and a power supply 1008.

[0209] Peripheral device interface 1003 can be used to connect at least one I / O (Input / Output) related peripheral device to processor 1001 and memory 1002. In some embodiments, processor 1001, memory 1002 and peripheral device interface 1003 are integrated on the same chip or circuit board; in some other embodiments, any one or two of processor 1001, memory 1002 and peripheral device interface 1003 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0210] The radio frequency (RF) circuit 1004 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF circuit 1004 communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 1004 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals back into electrical signals. Optionally, the RF circuit 1004 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, etc. The RF circuit 1004 can communicate with other terminals via at least one wireless communication protocol. This wireless communication protocol includes, but is not limited to: the World Wide Web, metropolitan area networks, intranets, various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks, and / or WiFi (Wireless Fidelity) networks. In some embodiments, the RF circuit 1004 may also include circuitry related to NFC (Near Field Communication), which is not limited in this application.

[0211] Display screen 1005 is used to display a UI (User Interface). This UI may include graphics, text, icons, videos, and any combination thereof. When display screen 1005 is a touch display screen, it also has the ability to collect touch signals on or above its surface. These touch signals can be input as control signals to processor 1001 for processing. In this case, display screen 1005 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, display screen 1005 may be a single screen, disposed on the front panel of terminal device 1000; in other embodiments, display screen 1005 may be at least two, disposed on different surfaces of terminal device 1000 or in a folded design; in still other embodiments, display screen 1005 may be a flexible display screen, disposed on a curved or folded surface of terminal device 1000. Furthermore, display screen 1005 may also be configured as a non-rectangular, irregular shape, i.e., a non-rectangular screen. The display screen 1005 can be made of materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).

[0212] The camera assembly 1006 is used to acquire images or videos. Optionally, the camera assembly 1006 includes a front-facing camera and a rear-facing camera. Typically, the front-facing camera is located on the front panel of the terminal, and the rear-facing camera is located on the back of the terminal. In some embodiments, there are at least two rear-facing cameras, which are any one of a main camera, a depth-sensing camera, a wide-angle camera, and a telephoto camera, to achieve background blurring by fusion of the main camera and the depth-sensing camera, panoramic shooting by fusion of the main camera and the wide-angle camera, VR (Virtual Reality) shooting, or other fusion shooting functions. In some embodiments, the camera assembly 1006 may also include a flash. The flash can be a single-color temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm-light flash and a cool-light flash, which can be used for light compensation at different color temperatures.

[0213] The audio circuit 1007 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, converting the sound waves into electrical signals that are input to the processor 1001 for processing, or input to the radio frequency circuit 1004 for voice communication. For stereo sound acquisition or noise reduction purposes, multiple microphones may be used, each located at a different part of the terminal device 1000. The microphone may also be an array microphone or an omnidirectional microphone. The speaker is used to convert electrical signals from the processor 1001 or the radio frequency circuit 1004 into sound waves. The speaker may be a conventional diaphragm speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can convert electrical signals not only into audible sound waves but also into inaudible sound waves for purposes such as distance measurement. In some embodiments, the audio circuit 1007 may also include a headphone jack.

[0214] The power supply 1008 is used to power the various components in the terminal device 1000. The power supply 1008 can be AC ​​power, DC power, a disposable battery, or a rechargeable battery. When the power supply 1008 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery that is charged via a wired line, while a wireless rechargeable battery is a battery that is charged via a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0215] In some embodiments, the terminal device 1000 further includes one or more sensors 1009. The one or more sensors 1009 include, but are not limited to: an acceleration sensor 1011, a gyroscope sensor 1012, a pressure sensor 1013, an optical sensor 1014, and a proximity sensor 1015.

[0216] Accelerometer 1011 can detect the magnitude of acceleration along the three coordinate axes of a coordinate system established by terminal device 1000. For example, accelerometer 1011 can be used to detect the components of gravitational acceleration along the three coordinate axes. Processor 1001 can control display screen 1005 to display the user interface in either a landscape or portrait view based on the gravitational acceleration signal acquired by accelerometer 1011. Accelerometer 1011 can also be used for games or for acquiring user motion data.

[0217] The gyroscope sensor 1012 can detect the orientation and rotation angle of the terminal device 1000. The gyroscope sensor 1012 can work in conjunction with the accelerometer sensor 1011 to collect the user's 3D movements on the terminal device 1000. Based on the data collected by the gyroscope sensor 1012, the processor 1001 can perform the following functions: motion sensing (e.g., changing the UI based on the user's tilt), image stabilization during shooting, game control, and inertial navigation.

[0218] The pressure sensor 1013 can be disposed on the side bezel of the terminal device 1000 and / or on the lower layer of the display screen 1005. When the pressure sensor 1013 is disposed on the side bezel of the terminal device 1000, it can detect the user's grip signal on the terminal device 1000, and the processor 1001 can perform left / right hand recognition or quick operation based on the grip signal collected by the pressure sensor 1013. When the pressure sensor 1013 is disposed on the lower layer of the display screen 1005, the processor 1001 can control the operable controls on the UI interface based on the user's pressure operation on the display screen 1005. The operable controls include at least one of button controls, scroll bar controls, icon controls, and menu controls.

[0219] An optical sensor 1014 is used to collect ambient light intensity. In one embodiment, the processor 1001 can control the display brightness of the display screen 1005 based on the ambient light intensity collected by the optical sensor 1014. Specifically, when the ambient light intensity is high, the display brightness of the display screen 1005 is increased; when the ambient light intensity is low, the display brightness of the display screen 1005 is decreased. In another embodiment, the processor 1001 can also dynamically adjust the shooting parameters of the camera assembly 1006 based on the ambient light intensity collected by the optical sensor 1014.

[0220] The proximity sensor 1015, also known as a distance sensor, is typically installed on the front panel of the terminal device 1000. The proximity sensor 1015 is used to detect the distance between the user and the front of the terminal device 1000. In one embodiment, when the proximity sensor 1015 detects that the distance between the user and the front of the terminal device 1000 is gradually decreasing, the processor 1001 controls the display screen 1005 to switch from a screen-on state to a screen-off state; when the proximity sensor 1015 detects that the distance between the user and the front of the terminal device 1000 is gradually increasing, the processor 1001 controls the display screen 1005 to switch from a screen-off state to a screen-on state.

[0221] Those skilled in the art will understand that Figure 10 The structure shown does not constitute a limitation on the terminal device 1000, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0222] Figure 11This is a schematic diagram of the server structure provided in the embodiments of this application. The server 1100 can vary considerably due to different configurations or performance. It may include one or more processors 1101 and one or more memories 1102. The one or more memories 1102 store at least one computer program, which is loaded and executed by the one or more processors 1101 to implement the facial model processing method provided in the above-described method embodiments. For example, the processor 1101 is a CPU. Of course, the server 1100 may also have wired or wireless network interfaces, a keyboard, and input / output interfaces for input and output. The server 1100 may also include other components for implementing device functions, which will not be elaborated here.

[0223] In an exemplary embodiment, a computer-readable storage medium is also provided, which stores at least one computer program that is loaded and executed by a processor to enable an electronic device to implement any of the above-described methods for processing facial models.

[0224] Optionally, the aforementioned computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, floppy disk, and optical data storage device, etc.

[0225] In an exemplary embodiment, a computer program or computer program product is also provided, which stores at least one computer program that is loaded and executed by a processor to enable an electronic device to implement any of the above-described methods for processing facial models.

[0226] It should be understood that "multiple" as used in this article refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0227] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0228] The above description is merely an exemplary embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the principles of this application should be included within the protection scope of this application.

Claims

1. A method for processing a facial model, characterized in that, The method includes: Obtain a reference facial image, which includes a wrinkle image region; The wrinkle image region is divided into blocks to obtain multiple wrinkle image blocks; For any wrinkle image patch, the feature processing model determines the features of the corresponding texture image patch based on the wrinkle image patch, where the texture image patch is an image patch in the permutation texture; the inverse transformation is performed on the features of the texture image patch to obtain the texture image patch; The texture image blocks corresponding to each wrinkle image block are stitched together to obtain the stitched area; The splicing area is processed to obtain the edge processing result; The target displacement map is determined based on the edge processing results; Convert the target displacement map into a target wrinkle map; In the facial model to be processed, a model region is determined, which is the region on the facial model to be processed that corresponds to the target wrinkle texture. The target wrinkle texture is fused onto the model area to obtain a facial model with wrinkles.

2. The method according to claim 1, characterized in that, After obtaining the reference facial image, the process further includes: The reference facial image is filtered to obtain the filtering result; Based on the filtering results, determine the probability that each pixel in the reference facial image belongs to the pixel corresponding to the wrinkle; The wrinkle image region is determined based on the probability that each pixel in the reference facial image belongs to the pixel corresponding to the wrinkle.

3. The method according to claim 2, characterized in that, The filtering process on the reference facial image to obtain the filtering result includes: The reference facial image is subjected to differential filtering to obtain the differential filtering result; The differential filtering result is then subjected to average filtering to obtain the average filtering result.

4. The method according to claim 1, characterized in that, The process of dividing the wrinkle image region into blocks to obtain multiple wrinkle image blocks includes: The wrinkle image region is divided into blocks based on multiple block windows to obtain each initial image block; The rotation parameters of each block window are determined based on each initial image block; Based on the rotation parameters of each block window, the block windows are rotated to obtain each rotated block window. The wrinkle image region is divided into blocks based on the rotated block windows to obtain individual wrinkle image blocks.

5. The method according to claim 4, characterized in that, The step of determining the rotation parameters of each segmented window based on each initial image block includes: For any initial image block, based on the pixel information of each pixel in the initial image block, determine the gradient of each pixel in the horizontal direction and the gradient of each pixel in the vertical direction. Based on the gradient of each pixel in the horizontal direction and the gradient of each pixel in the vertical direction in any initial image block, the rotation parameters of the block window corresponding to the any initial image block are determined.

6. The method according to claim 5, characterized in that, The rotation parameters include the rotation angle; The step of determining the rotation parameters of the block window corresponding to any initial image block based on the gradient of each pixel in the horizontal direction and the gradient of each pixel in the vertical direction of any initial image block includes: Determine the ratio between a first cumulative value and a second cumulative value, wherein the first cumulative value is the sum of the absolute values ​​of the gradients of each pixel in the horizontal direction in any initial image block, and the second cumulative value is the sum of the absolute values ​​of the gradients of each pixel in the vertical direction in any initial image block; The rotation angle of the block window corresponding to any initial image block is determined based on the ratio.

7. The method according to claim 5, characterized in that, The rotation parameters include the rotation direction; The step of determining the rotation parameters of the block window corresponding to any initial image block based on the gradient of each pixel in the horizontal direction and the gradient of each pixel in the vertical direction of any initial image block includes: For any pixel in any initial image block, determine the maximum value corresponding to the pixel. The maximum value corresponding to the pixel is the maximum value between the absolute value of the gradient of the pixel in the horizontal direction and the absolute value of the gradient of the pixel in the vertical direction. Based on the maximum value of each pixel in any initial image block, the rotation direction of the block window corresponding to any initial image block is determined.

8. The method according to claim 1, characterized in that, The feature processing model includes at least one cascaded random fern classifier; The step of determining the features of the texture image block corresponding to any wrinkle image block based on the feature processing model includes: Obtain a first initial feature, which is input information set for the feature processing model; The first initial feature is input into the at least one cascaded random fern classifier, and the at least one cascaded random fern classifier is controlled to modify the first initial feature multiple times based on any wrinkle image patch to obtain the feature of the texture image patch corresponding to any wrinkle image patch.

9. The method according to claim 1, characterized in that, Before determining the features of the texture image block corresponding to any wrinkle image block based on the feature processing model, the method further includes: Obtain a sample facial image and a sample permutation map corresponding to the sample facial image, wherein the sample facial image includes the sample image region where wrinkles are located, and the sample permutation map includes the sample map region where wrinkles are located; The sample image region is divided into blocks to obtain each first image block; The sample mapping area is divided into blocks to obtain individual second image blocks; The neural network model is trained based on the first image blocks and the second image blocks to obtain the feature processing model.

10. The method according to claim 9, characterized in that, One first image block corresponds to one second image block; The step of training a neural network model based on each of the first image patches and each of the second image patches to obtain the feature processing model includes: Obtain a second initial feature, which is the input information set for the neural network model; For any first image patch, the neural network model is controlled to modify the second initial feature according to the first image patch to obtain the first feature of the second image patch corresponding to the first image patch; Extract the second features of each of the second image blocks; Based on the first and second features of each of the second image blocks, the neural network model is adjusted to obtain the feature processing model.

11. The method according to claim 10, characterized in that, The neural network model includes at least one cascaded initial classifier; The process of controlling the neural network model to modify the second initial feature based on any first image patch to obtain the first feature of the second image patch corresponding to any first image patch includes: The second initial feature is input into the at least one cascaded initial classifier to obtain the output information of each initial classifier. The output information of the previous initial classifier in the at least one cascaded initial classifier is the input information of the next initial classifier, and the output information of the last initial classifier is the first feature of the second image block corresponding to any first image block. The step of adjusting the neural network model based on the first and second features of each of the second image patches to obtain the feature processing model includes: For any one of the at least one cascaded initial classifiers, the loss value of the initial classifier is determined based on the input information of the initial classifier, the second features of each second image patch, and the output information of the initial classifier. Based on the loss values ​​of each initial classifier in the at least one cascaded initial classifier, the neural network model is adjusted to obtain the feature processing model.

12. A facial model processing device, characterized in that, The device includes: The acquisition module is used to acquire a reference facial image, which includes a wrinkle image region; The segmentation module is used to segment the wrinkle image region into multiple wrinkle image blocks. The determination module is used to, for any wrinkle image patch, determine the features of the corresponding texture image patch based on the wrinkle image patch using a feature processing model, wherein the texture image patch is an image patch in a displacement map; perform an inverse transform on the features of the texture image patch to obtain the texture image patch; stitch together the texture image patches corresponding to each wrinkle image patch to obtain a stitched region; perform edge processing on the stitched region to obtain an edge processing result; determine a target displacement map based on the edge processing result; and convert the target displacement map into a target wrinkle map. The determining module is used to determine a model region in the face model to be processed, wherein the model region is the region on the face model to be processed that corresponds to the target wrinkle texture. The fusion module is used to fuse the target wrinkle texture onto the model area to obtain a facial model with wrinkles.

13. The apparatus according to claim 12, characterized in that, The device further includes: The filtering module is used to filter the reference facial image to obtain the filtering result; The determining module is further configured to determine, based on the filtering result, the probability that each pixel in the reference facial image belongs to the pixel corresponding to the wrinkle; and to determine the wrinkle image region based on the probability that each pixel in the reference facial image belongs to the pixel corresponding to the wrinkle.

14. The apparatus according to claim 13, characterized in that, The filtering module is used to perform differential filtering on the reference facial image to obtain a differential filtering result; and to perform average filtering on the differential filtering result to obtain an average filtering result.

15. The apparatus according to claim 12, characterized in that, The segmentation module is used to segment the wrinkle image region into blocks based on multiple segmentation windows to obtain each initial image block; and to determine the rotation parameters of each segmentation window based on each initial image block. The segmented windows are rotated based on their rotation parameters to obtain rotated segmented windows; the wrinkle image region is then segmented based on the rotated segmented windows to obtain wrinkle image blocks.

16. The apparatus according to claim 15, characterized in that, The segmentation module is used to, for any initial image block, determine the gradient of each pixel in the horizontal direction and the gradient of each pixel in the vertical direction based on the pixel information of each pixel in the initial image block; and determine the rotation parameters of the segmented window corresponding to the initial image block based on the gradient of each pixel in the horizontal direction and the gradient of each pixel in the vertical direction.

17. The apparatus according to claim 16, characterized in that, The rotation parameters include the rotation angle; The segmentation module is used to determine the ratio between a first cumulative value and a second cumulative value, wherein the first cumulative value is the sum of the absolute values ​​of the gradients of each pixel in the horizontal direction in any initial image block, and the second cumulative value is the sum of the absolute values ​​of the gradients of each pixel in the vertical direction in any initial image block; and to determine the rotation angle of the segmentation window corresponding to the any initial image block based on the ratio.

18. The apparatus according to claim 16, characterized in that, The rotation parameters include the rotation direction; The segmentation module is used to determine the maximum value corresponding to any pixel in any initial image block, wherein the maximum value corresponding to any pixel is the maximum value between the absolute value of the gradient of the pixel in the horizontal direction and the absolute value of the gradient of the pixel in the vertical direction; and to determine the rotation direction of the segmentation window corresponding to the initial image block based on the maximum values ​​corresponding to each pixel in the initial image block.

19. The apparatus according to claim 12, characterized in that, The feature processing model includes at least one cascaded random fern classifier; The determining module is used to obtain a first initial feature, which is input information set for the feature processing model; input the first initial feature into the at least one cascaded random fern classifier, and control the at least one cascaded random fern classifier to perform multiple corrections on the first initial feature based on any wrinkle image patch to obtain the feature of the texture image patch corresponding to any wrinkle image patch.

20. The apparatus according to claim 12, characterized in that, The acquisition module is further configured to acquire a sample face image and a sample displacement map corresponding to the sample face image, wherein the sample face image includes a sample image region where wrinkles are located, and the sample displacement map includes a sample map region where wrinkles are located. The segmentation module is further configured to segment the sample image region into blocks to obtain each first image block; and to segment the sample texture region into blocks to obtain each second image block. The acquisition module is further configured to train the neural network model based on the first image blocks and the second image blocks to obtain the feature processing model.

21. The apparatus according to claim 20, characterized in that, One first image block corresponds to one second image block; The acquisition module is used to acquire a second initial feature, which is input information set for the neural network model; for any first image block, the neural network model is controlled to modify the second initial feature according to the first image block to obtain a first feature of the second image block corresponding to the first image block; the second feature of each second image block is extracted; based on the first feature and the second feature of each second image block, the neural network model is adjusted to obtain the feature processing model.

22. The apparatus according to claim 21, characterized in that, The neural network model includes at least one cascaded initial classifier; The acquisition module is used to input the second initial feature into the at least one cascaded initial classifier to obtain the output information of each initial classifier. The output information of the previous initial classifier in the at least one cascaded initial classifier is the input information of the next initial classifier, and the output information of the last initial classifier is the first feature of the second image block corresponding to any first image block. The acquisition module is used to determine the loss value of any initial classifier among the at least one cascaded initial classifiers, based on the input information of the initial classifier, the second features of each second image patch, and the output information of the initial classifier; and to adjust the neural network model based on the loss values ​​of each initial classifier among the at least one cascaded initial classifiers to obtain the feature processing model.

23. An electronic device, characterized in that, The electronic device includes a processor and a memory, the memory storing at least one computer program, the at least one computer program being loaded and executed by the processor to enable the electronic device to implement the facial model processing method as described in any one of claims 1 to 11.

24. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one computer program, which is loaded and executed by a processor to enable an electronic device to implement the facial model processing method as described in any one of claims 1 to 11.

25. A computer program product, characterized in that, The computer program product stores at least one computer program, which is loaded and executed by a processor to enable the electronic device to implement the facial model processing method as described in any one of claims 1 to 11.

Images