Image processing method and device, electronic equipment and storage medium

Abstract

The invention relates to an image processing method and device, electronic equipment and a storage medium. The method comprises the following steps: performing recognition processing on each frame of video image of a video stream to obtain an initial image containing a human face; performing alignment processing on a face region in the initial image and a preset skin template to obtain a mapping relation between the face region and the preset skin template, and adjusting a skin region of the face region to obtain an adjusted image; the skin area is determined according to the mapping relation; performing image quality enhancement processing on the adjusted image to obtain an enhanced image; and rendering the enhanced image to obtain a target image. According to the embodiment, through the face adjustment processing and the image quality enhancement processing, the quality of the target image can be improved, and the user experience can be improved.

Description

Image processing method and device, electronic device, and storage medium Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to an image processing method and device, an electronic device, and a storage medium. Background Art

[0002] As communication needs increase, more and more users are choosing online meetings, such as video conferencing and voice conferencing. Taking video conferencing as an example, electronic devices can capture user images and upload them to the conference room, allowing each user to see other users, achieving a similar effect to an in-person meeting.

[0003] However, related technologies usually display the user image directly, so that other users can see the user's facial details, such as forehead texture or spots, etc., which reduces the user experience.

[0004] Summary of the Invention

[0005] The present disclosure provides an image processing method and apparatus, an electronic device, and a storage medium to address the deficiencies of related technologies.

[0006] According to a first aspect of an embodiment of the present disclosure, there is provided an image processing method, the method comprising:

[0007] Perform recognition processing on each frame of the video stream to obtain an initial image containing a human face;

[0008] Aligning the face region in the initial image with a preset skin template to obtain a mapping relationship between the face region and the preset skin template;

[0009] Adjusting the skin area of ​​the face area to obtain an adjusted image; the skin area is determined according to the mapping relationship;

[0010] Performing image quality enhancement processing on the adjusted image to obtain an enhanced image;

[0011] The enhanced image is rendered to obtain a target image.

[0012] Optionally, aligning the face region in the initial image with a preset skin template to obtain a mapping relationship between the face region and the preset skin template includes:

[0013] Obtaining the key point coordinates of the face in the preset skin template to obtain a first key point coordinate vector;

[0014] Obtaining the coordinates of key points of the face in the initial image to obtain a second key point coordinate vector;

[0015] The first key point coordinate vector and the second key point coordinate vector are input into a preset distance model and a solution of the preset distance model is obtained, and the solution of the preset distance model is used as the mapping relationship.

[0016] Optionally, obtaining a solution to the preset distance model includes:

[0017] Respectively obtaining a coordinate mean and a coordinate standard deviation of the first key point coordinate vector to obtain a first coordinate mean and a first coordinate standard deviation;

[0018] Respectively obtaining a coordinate mean and a coordinate standard deviation of the coordinate vector of the second key point to obtain a second coordinate mean and a second coordinate standard deviation;

[0019] Subtract the first coordinate mean from the first key point coordinate vector and divide the result by the first coordinate standard deviation to obtain a first adjusted coordinate vector;

[0020] Subtract the second coordinate mean from the second key point coordinate vector and divide the result by the second coordinate standard deviation to obtain a second adjusted coordinate vector;

[0021] The first adjusted coordinate vector and the second adjusted coordinate vector are input into the preset distance model and singular value decomposition is performed on the preset distance model to obtain the mapping relationship.

[0022] Optionally, adjusting the skin area of ​​the face area to obtain an adjusted image includes:

[0023] In response to successively scaling the initial image to a preset size ratio to obtain a candidate size image, performing a clearing process on the candidate size image to obtain a processed image;

[0024] In response to the number of resizing operations being equal to a preset number threshold, the final processed image is restored to the size of the initial image to obtain a first candidate image, and the first candidate image is used as the adjusted image.

[0025] Optionally, performing a clearing process on the candidate size image to obtain a processed image includes:

[0026] Detecting at least one spot region in the skin region of the candidate size image; the spot region refers to a connected domain having an area within a first preset area range and a size smaller than a first preset size threshold and larger than a second preset size threshold;

[0027] performing a clearing process on the at least one spot region to obtain a first processed image;

[0028] In response to satisfying the first loop condition, restoring the first processed image to the size of the initial image to obtain a final processed image;

[0029] The first cycle condition means that the number of times the spot areas are detected is equal to a first preset number threshold or the number of the spot areas is less than the first preset number threshold.

[0030] Optionally, detecting at least one spot area in the skin area of ​​the candidate size image includes:

[0031] Acquire target pixel data according to the mapping relationship and pixel data of each pixel point of the preset skin template;

[0032] Replacing pixel data at a corresponding position in the candidate size image with the target pixel data to obtain a second candidate image;

[0033] Connected domain detection is performed on the skin area of ​​the second candidate image to obtain a connected domain whose spot area size in the skin area is smaller than a first preset size threshold and larger than a second preset size threshold as the spot area of ​​the skin area.

[0034] Optionally, performing connected domain detection on the skin area of ​​the second candidate image to obtain a spot area of ​​the skin area includes:

[0035] Converting the grayscale image into a binary image according to each preset threshold value to obtain multiple frames of binary images;

[0036] The connected areas of the binary images of each frame are obtained respectively, and multiple connected domains of the binary images of each frame are obtained;

[0037] Classifying the multiple connected domains according to the distances between the center points of each connected domain of the multiple frames of binary images, and obtaining multiple connected domains whose distances between the center points are less than a preset distance threshold;

[0038] The positions and sizes of the multiple connected domains are obtained, and the connected domains whose sizes are smaller than a first preset size threshold and larger than a second preset size threshold are used as the spot areas of the skin area; the position refers to the average value of the center point coordinates of the connected domains of each frame of the binary image; the size refers to the radius length of the circumscribed circle when the area of ​​the connected domain of the binary image is within a preset area range.

[0039] Optionally, after detecting at least one spot area in the skin area of ​​the candidate size image, the method further includes:

[0040] In response to not detecting the spot area, determining whether a stripe area exists; wherein the stripe area refers to a connected domain having an area within a second preset area range, a size within a preset length range, and a width within a preset width range;

[0041] In response to detecting at least one stripe region, clearing the stripe region that meets a second preset condition in the candidate size image to obtain a second processed image;

[0042] In response to satisfying the second loop condition, restoring the second processed image to the size of the initial image to obtain a final processed image;

[0043] The second cycle condition means that the number of times the stripe regions are detected is equal to a second preset number threshold or the number of the stripe regions is less than a second preset number threshold.

[0044] Optionally, performing a clearing process on the candidate size image to obtain a processed image includes:

[0045] Detecting a stripe region of the skin area of ​​the candidate size image; wherein the stripe region refers to a connected domain having an area within a second preset area range, a size within a preset length range, and a width within a preset width range;

[0046] In response to detecting at least one stripe region, clearing the stripe region that meets a second preset condition in the candidate size image to obtain a third processed image;

[0047] In response to satisfying the second loop condition, restoring the third processed image to the size of the initial image to obtain a final processed image;

[0048] The second cycle condition means that the number of times the stripe regions are detected is equal to a second preset number threshold or the number of the stripe regions is less than a second preset number threshold.

[0049] Optionally, performing face adjustment processing on the face region in the initial image to obtain an adjusted image includes:

[0050] Aligning the face region in the initial image with a preset skin template to obtain a mapping relationship between the face region and the preset skin template;

[0051] In response to the initial image being sequentially scaled into candidate size images of different preset sizes, an abnormal region of a skin area in the candidate size image is obtained to obtain an abnormal region feature map; the abnormal region includes a spot region and / or a stripe region; the spot region refers to a connected domain having an area within a first preset area range, a size smaller than a first preset size threshold and larger than a second preset size threshold; the stripe region refers to a connected domain having an area within a second preset area range, a size within a preset length range, and a width within a preset width range; the second preset area range is larger than the first preset area range;

[0052] In response to the number of resizing operations being equal to a preset number threshold, the abnormal region of the last candidate size image is cleared to obtain the adjusted image.

[0053] Optionally, after acquiring the first candidate image, the method further includes:

[0054] The eye region in the first candidate image is scaled to obtain a second candidate image, and the second candidate image is used as the adjusted image.

[0055] Optionally, performing scaling processing on the eye region in the first candidate image to obtain a second candidate image includes:

[0056] Obtaining a zoom ratio of each pixel in the eye area of ​​the first candidate image;

[0057] Acquire target coordinate data of each pixel point according to the scaling ratio and the original coordinate data of each pixel point;

[0058] The original coordinate data of each pixel in the first candidate image is replaced with the target coordinate data to obtain the second candidate image.

[0059] Optionally, obtaining a scaling ratio of each pixel in the eye region of the first candidate image includes:

[0060] Obtaining a distance ratio of each pixel point in the eye area of ​​the first candidate image; the distance ratio is used to indicate that the closer the pixel point is to the center point of the eye area, the smaller the moving distance ratio;

[0061] A scaling ratio of each pixel point is obtained according to the distance ratio; the scaling ratio is proportional to the distance ratio.

[0062] Optionally, obtaining the distance ratio of each pixel point in the eye area of ​​the first candidate image includes:

[0063] Obtaining the major axis length and minor axis length of the eye region;

[0064] The distance ratio of each pixel point is obtained according to the long axis length, the short axis length, the coordinate data of the center point of the eye area and the original coordinate data of each pixel point.

[0065] Optionally, obtaining a zoom ratio of each pixel point according to the distance ratio includes:

[0066] The scaling ratio of each pixel point is obtained according to the distance ratio and the initial value of the scaling ratio.

[0067] Optionally, after acquiring the second candidate image, the method further includes:

[0068] The face region in the second candidate image is resized to obtain a third candidate image, and the third candidate image is used as the adjusted image.

[0069] Optionally, resizing the face region in the second candidate image to obtain a third candidate image includes:

[0070] Obtaining the distance between the control point C1 of the face area and each pixel point in the face area;

[0071] Obtaining a deformation ratio of each pixel point according to the distance value, wherein the deformation ratio is used to indicate that the farther the pixel point is from the control point C, the smaller the movement distance;

[0072] Calculating target pixel data of each pixel point based on the coordinate data of the target position M of the control point C, the coordinate data of the control point C, and the coordinate data of the target position X of each pixel point;

[0073] The pixel points of the face area of ​​the second candidate image are replaced with the target pixel data to obtain the third candidate image.

[0074] Optionally, the image quality enhancement includes skin smoothing, whitening, and brightening, and performing image quality enhancement on the adjusted image to obtain an enhanced image includes:

[0075] performing skin resurfacing processing on the adjusted image to obtain a first skin resurfacing feature map;

[0076] adjusting the skin resurfacing degree of the skin resurfacing feature map to obtain a second skin resurfacing feature map;

[0077] performing skin color brightening processing on the second skin resurfacing feature map to obtain a skin color brightening feature map;

[0078] Performing soft light processing based on the skin color brightening feature map and the adjusted image to obtain a soft light feature map;

[0079] The enhanced image is obtained according to the adjusted image, the second skin resurfacing feature map, the skin color brightening feature map, the soft light feature map, and their respective weight values.

[0080] Optionally, performing skin resurfacing processing on the adjusted image to obtain a first skin resurfacing feature map includes:

[0081] performing filtering processing on the green sub-image of the adjusted image to obtain a filtered sub-image;

[0082] Obtaining a difference in pixel data of a same pixel point in the adjusted image and the filtered sub-image to obtain a difference sub-image;

[0083] Compensating pixel data of each pixel point in the difference sub-image based on preset compensation data to obtain the difference-compensated sub-image;

[0084] Light compensation is performed on the difference-compensated sub-image to obtain the first skin resurfacing feature map.

[0085] Optionally, the filtering method used for filtering the green sub-image of the adjusted image includes one of the following: averaging filtering, Gaussian filtering, bilateral filtering and guided filtering.

[0086] Optionally, the filter kernel of the Gaussian filter is implemented using a circular filter kernel, and the sum of the weight values ​​of the circular filter kernel is equal to a preset weight value.

[0087] Optionally, performing light compensation on the difference-compensated sub-image to obtain the first skin resurfacing feature map includes:

[0088] Comparing pixel data of each pixel point of the difference-compensated sub-image with preset compensation data;

[0089] When the pixel data of each pixel point is greater than or equal to the preset compensation data, obtaining target pixel data corresponding to the pixel data according to the first light compensation model;

[0090] When the pixel data of each pixel point is less than the preset compensation data, obtaining target pixel data corresponding to the pixel data according to a second light compensation model;

[0091] Replacing pixel data of each pixel point of the difference-compensated sub-image with the target pixel data;

[0092] In response to the number of compensations reaching a preset compensation threshold, the difference-compensated sub-image obtained by the last compensation is determined as the first skin resurfacing feature image.

[0093] Optionally, the preset compensation times threshold value has a value range of [1, 10].

[0094] Optionally, adjusting the skin resurfacing degree of the skin resurfacing feature map to obtain a second skin resurfacing feature map includes:

[0095] Obtaining a grayscale image corresponding to the adjusted image;

[0096] Brightening the grayscale data of each pixel in the grayscale image based on the preset skin resurfacing degree data to obtain a grayscale brightened image;

[0097] Pixel data of each pixel point of the adjusted image is adjusted according to the first skin resurfacing feature map and the grayscale brightened image to obtain the second skin resurfacing feature map.

[0098] Optionally, after performing recognition processing on each frame of the video image of the video stream, the method further includes:

[0099] In response to no human face region being detected in the video image, performing image quality enhancement processing on the video image to obtain an enhanced image;

[0100] The enhanced image is rendered to obtain a target image.

[0101] According to a second aspect of an embodiment of the present disclosure, there is provided an image processing apparatus, the apparatus comprising:

[0102] The initial image acquisition module is used to identify and process each frame of the video stream to obtain an initial image containing a human face;

[0103] a mapping relationship acquisition module, configured to align the face region in the initial image with a preset skin template to obtain a mapping relationship between the face region and the preset skin template;

[0104] an adjusted image acquisition module, configured to adjust the skin area of ​​the face area to obtain an adjusted image; the skin area is determined according to the mapping relationship;

[0105] An enhanced image acquisition module, configured to perform image quality enhancement processing on the adjusted image to obtain an enhanced image;

[0106] The target image acquisition module is used to render the enhanced image to obtain a target image.

[0107] Optionally, the mapping relationship acquisition module includes:

[0108] A first vector acquisition submodule is used to obtain the key point coordinates of the face in the preset skin template to obtain a first key point coordinate vector;

[0109] A second vector acquisition submodule is used to obtain the key point coordinates of the face in the initial image to obtain a second key point coordinate vector;

[0110] A mapping relationship acquisition submodule is used to input the first key point coordinate vector and the second key point coordinate vector into a preset distance model and obtain a solution of the preset distance model, and use the solution of the preset distance model as the mapping relationship.

[0111] Optionally, the mapping relationship acquiring unit includes:

[0112] A first vector processing unit, configured to respectively obtain a coordinate mean and a coordinate standard deviation of the first key point coordinate vector to obtain a first coordinate mean and a first coordinate standard deviation;

[0113] A second vector processing unit, configured to respectively obtain a coordinate mean and a coordinate standard deviation of the second key point coordinate vector to obtain a second coordinate mean and a second coordinate standard deviation;

[0114] a first vector adjustment unit, configured to subtract the first coordinate mean from the first key point coordinate vector and divide the result by the first coordinate standard deviation to obtain a first adjusted coordinate vector;

[0115] a second vector adjustment unit, configured to subtract the second coordinate mean from the second key point coordinate vector and divide the resultant value by the second coordinate standard deviation to obtain a second adjusted coordinate vector;

[0116] A mapping relationship acquisition unit is configured to input the first adjusted coordinate vector and the second adjusted coordinate vector into the preset distance model and perform singular value decomposition on the preset distance model to obtain the mapping relationship.

[0117] Optionally, the adjustment image acquisition module includes:

[0118] a processed image acquisition submodule, configured to, in response to scaling the initial image in sequence to candidate size images of preset size ratios, perform clearing processing on the candidate size images to obtain processed images;

[0119] The first candidate image acquisition submodule is configured to restore the final processed image to the size of the initial image in response to the number of resizing operations being equal to a preset number threshold, to obtain a first candidate image, and use the first candidate image as the adjusted image.

[0120] Optionally, the processing image acquisition submodule includes:

[0121] a spot region detection unit, configured to detect at least one spot region in the skin region of the candidate size image; the spot region refers to a connected domain having an area within a first preset area range and a size smaller than a first preset size threshold and larger than a second preset size threshold;

[0122] a first processed image acquisition unit, configured to perform a clearing process on the at least one spot region to obtain a first processed image;

[0123] a processed image acquisition unit, configured to restore the first processed image to the size of the initial image in response to satisfying the first loop condition, to obtain a final processed image;

[0124] The first cycle condition means that the number of times the spot areas are detected is equal to a first preset number threshold or the number of the spot areas is less than the first preset number threshold.

[0125] Optionally, the spot area detection unit includes:

[0126] a target pixel acquisition subunit, configured to acquire target pixel data according to the mapping relationship and the pixel data of each pixel point of the preset skin template;

[0127] A second candidate image acquisition subunit is configured to replace pixel data at a corresponding position in the candidate size image with the target pixel data to obtain a second candidate image;

[0128] The spot area acquisition subunit is used to perform connected domain detection on the skin area of ​​the second candidate image, and obtain a connected domain whose spot area size of the skin area is smaller than a first preset size threshold and larger than a second preset size threshold as the spot area of ​​the skin area.

[0129] Optionally, the spot area acquisition subunit includes:

[0130] A binary image acquisition subunit is used to convert the grayscale image into a binary image according to each preset threshold value to obtain multiple frames of binary images;

[0131] A first connected domain acquisition subunit is used to respectively acquire connected areas of the binary image of each frame to obtain multiple connected domains of the binary image of each frame;

[0132] a second connected domain acquisition subunit, configured to classify the multiple connected domains according to the distances between the center points of the connected domains of the multiple binary image frames, and obtain multiple connected domains whose distances between the center points are less than a preset distance threshold;

[0133] The spot area acquisition subunit is used to obtain the positions and sizes of the multiple connected domains, and to use the connected domains whose sizes are smaller than a first preset size threshold and larger than a second preset size threshold as the spot areas of the skin area; the position refers to the average value of the center point coordinates of the connected domains of each frame of the binary image; the size refers to the radius length of the circumscribed circle when the area of ​​the connected domain of the binary image is within a preset area range.

[0134] Optionally, after detecting at least one spot area in the skin area of ​​the candidate size image, the processing image acquisition submodule further includes:

[0135] a stripe region determination unit, configured to determine whether a stripe region exists in response to no spot region being detected; wherein the stripe region refers to a connected domain having an area within a second preset area range, a size within a preset length range, and a width within a preset width range;

[0136] a spot area removing unit, configured to, in response to detecting at least one stripe area, remove the stripe area in the candidate size image that meets a second preset condition, to obtain a second processed image;

[0137] a processed image acquisition unit, configured to restore the second processed image to the size of the initial image in response to satisfying the second loop condition, to obtain a final processed image;

[0138] The second cycle condition means that the number of times the stripe regions are detected is equal to a second preset number threshold or the number of the stripe regions is less than a second preset number threshold.

[0139] Optionally, the processing image acquisition submodule includes:

[0140] a stripe region detection unit, configured to detect a stripe region in the skin region of the candidate size image; wherein the stripe region refers to a connected domain having an area within a second preset area range, a size within a preset length range, and a width within a preset width range;

[0141] a third processed image acquisition unit, configured to, in response to detecting at least one stripe region, remove the stripe region in the candidate size image that meets a second preset condition, to obtain a third processed image;

[0142] a processed image acquisition unit, configured to restore the third processed image to the size of the initial image in response to satisfying the second loop condition, to obtain a final processed image;

[0143] The second cycle condition means that the number of times the stripe regions are detected is equal to a second preset number threshold or the number of the stripe regions is less than a second preset number threshold.

[0144] Optionally, the adjustment image acquisition module includes:

[0145] a mapping relationship acquisition submodule, configured to align the face region in the initial image with a preset skin template to obtain a mapping relationship between the face region and the preset skin template;

[0146] An abnormal region acquisition submodule is configured to, in response to the initial image being sequentially scaled into candidate size images of different preset sizes, acquire abnormal regions of the skin region in the candidate size images to obtain an abnormal region feature map; the abnormal regions include spot regions and / or stripe regions; the spot regions refer to connected domains whose areas are within a first preset area range, whose sizes are smaller than a first preset size threshold and larger than a second preset size threshold; the stripe regions refer to connected domains whose areas are within a second preset area range, whose sizes are within a preset length range, and whose widths are within a preset width range; the second preset area range is larger than the first preset area range;

[0147] The adjusted image acquisition submodule is configured to, in response to the number of resizing operations being equal to a preset number threshold, remove abnormal regions on the last candidate size image to obtain the adjusted image.

[0148] Optionally, the adjusted image acquisition module further includes:

[0149] The second candidate image acquisition submodule is configured to perform scaling processing on the eye region in the first candidate image to obtain a second candidate image, and use the second candidate image as the adjusted image.

[0150] Optionally, the second candidate image acquisition submodule includes:

[0151] a zoom ratio acquisition unit, configured to acquire a zoom ratio of each pixel in the eye area of ​​the first candidate image;

[0152] a target coordinate acquisition unit, configured to acquire target coordinate data of each pixel point according to the zoom ratio and the original coordinate data of each pixel point;

[0153] The second candidate image acquisition unit is configured to replace the original coordinate data of each pixel point in the first candidate image with the target coordinate data to obtain the second candidate image.

[0154] Optionally, the scaling ratio acquiring unit includes:

[0155] a distance ratio acquisition unit, configured to acquire a distance ratio of each pixel point in the eye region of the first candidate image; the distance ratio being used to indicate that the closer the pixel point is to the center point of the eye region, the smaller the moving distance ratio;

[0156] A zoom ratio acquisition unit is used to acquire the zoom ratio of each pixel point according to the distance ratio; the zoom ratio is proportional to the distance ratio.

[0157] Optionally, the distance ratio acquiring unit includes:

[0158] a length acquisition subunit, configured to acquire the major axis length and minor axis length of the eye region;

[0159] The distance ratio acquisition subunit is used to acquire the distance ratio of each pixel point according to the major axis length, the minor axis length, the coordinate data of the center point of the eye area and the original coordinate data of each pixel point.

[0160] Optionally, the scaling ratio acquiring unit includes:

[0161] The zoom ratio acquisition subunit is configured to acquire the zoom ratio of each pixel point according to the distance ratio and the initial value of the zoom ratio.

[0162] Optionally, the second candidate image acquisition submodule further includes:

[0163] The third candidate image acquisition unit is configured to resize the face region in the second candidate image to obtain a third candidate image, and use the third candidate image as the adjusted image.

[0164] Optionally, the third candidate image acquisition unit includes:

[0165] A distance value acquisition subunit, configured to acquire distance values ​​between the control point C1 of the face region and each pixel point in the face region;

[0166] a deformation ratio obtaining subunit, configured to obtain a deformation ratio of each pixel point according to the distance value, wherein the deformation ratio indicates that the farther the pixel point is from the control point C, the smaller the movement distance;

[0167] a target pixel acquisition subunit, configured to calculate target pixel data of each pixel point based on the coordinate data of the target position M of the control point C, the coordinate data of the control point C, and the coordinate data of the target position X of each pixel point;

[0168] The third candidate image acquisition subunit is configured to replace the pixel points of the face area of ​​the second candidate image with the target pixel data to obtain the third candidate image.

[0169] Optionally, the image quality enhancement includes skin resurfacing, whitening, and brightening, and the enhanced image acquisition module includes:

[0170] A first feature map acquisition submodule, configured to perform skin resurfacing processing on the adjusted image to obtain a first skin resurfacing feature map;

[0171] A second feature map acquisition submodule is used to adjust the skin resurfacing degree of the skin resurfacing feature map to obtain a second skin resurfacing feature map;

[0172] a skin color brightening feature map acquisition submodule, configured to perform skin color brightening processing on the second skin resurfacing feature map to obtain a skin color brightening feature map;

[0173] a soft light feature map acquisition submodule, configured to perform soft light processing based on the skin color brightening feature map and the adjusted image to obtain a soft light feature map;

[0174] The enhanced image acquisition submodule is used to acquire the enhanced image according to the adjusted image, the second skin resurfacing feature map, the skin color brightening feature map, the soft light feature map and their respective weight values.

[0175] Optionally, the first feature map acquisition submodule includes:

[0176] a filtered sub-image obtaining unit, configured to perform filtering processing on the green sub-image of the adjusted image to obtain a filtered sub-image;

[0177] a difference sub-image acquisition unit, configured to acquire a difference in pixel data of a same pixel point in the adjusted image and the filtered sub-image to obtain a difference sub-image;

[0178] a compensation sub-image acquisition unit, configured to compensate pixel data of each pixel point in the difference sub-image based on preset compensation data to obtain the difference compensation sub-image;

[0179] The first feature map acquisition unit is configured to perform light compensation on the difference compensation sub-image to obtain the first skin resurfacing feature map.

[0180] Optionally, the filtering method used for filtering the green sub-image of the adjusted image includes one of the following: averaging filtering, Gaussian filtering, bilateral filtering and guided filtering.

[0181] Optionally, the filter kernel of the Gaussian filter is implemented using a circular filter kernel, and the sum of the weight values ​​of the circular filter kernel is equal to a preset weight value.

[0182] Optionally, the first feature map acquiring unit includes:

[0183] a pixel data comparison subunit, configured to compare pixel data of each pixel point of the difference-compensated sub-image with preset compensation data;

[0184] a target pixel data acquisition subunit, configured to acquire target pixel data corresponding to the pixel data according to a first light compensation model when the pixel data of each pixel point is greater than or equal to the preset compensation data;

[0185] The target pixel data acquisition subunit is further configured to acquire target pixel data corresponding to the pixel data according to a second light compensation model when the pixel data of each pixel point is less than the preset compensation data;

[0186] a pixel data replacement subunit, configured to replace pixel data of each pixel point of the difference-compensated sub-image with the target pixel data;

[0187] The first feature map acquisition subunit is configured to determine, in response to the number of compensation times reaching a preset compensation times threshold, that the difference compensation sub-image obtained by the last compensation is the first skin resurfacing feature map.

[0188] Optionally, the preset compensation times threshold value has a value range of [1, 10].

[0189] Optionally, the second feature map acquisition submodule includes:

[0190] A grayscale image acquisition unit, configured to acquire a grayscale image corresponding to the adjusted image;

[0191] a brightened image acquisition unit, configured to perform a brightening process on the grayscale data of each pixel in the grayscale image based on preset skin resurfacing degree data to obtain a grayscale brightened image;

[0192] The second feature map acquiring unit is configured to adjust the pixel data of each pixel point of the adjusted image according to the first skin resurfacing feature map and the grayscale brightened image to obtain the second skin resurfacing feature map.

[0193] Optionally, after performing recognition processing on each frame of the video image of the video stream, the apparatus further includes:

[0194] The enhanced image acquisition module is further configured to, in response to no human face region being detected in the video image, perform image quality enhancement processing on the video image to obtain an enhanced image;

[0195] The target image acquisition module is further used to render the enhanced image to obtain a target image.

[0196] According to a third aspect of the embodiments of the present disclosure, there is provided an electronic device, comprising

[0197] Display screen;

[0198] processor;

[0199] a memory for storing a computer program executable by the processor;

[0200] The processor is configured to execute the computer program in the memory to implement the method as described in any one of the first aspects.

[0201] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, which, when an executable computer program in the storage medium is executed by a processor, can implement the method described in any one of the first aspects.

[0202] The technical solutions provided by the embodiments of the present disclosure may have the following beneficial effects:

[0203] As can be seen from the above embodiments, the solution provided by the embodiments of the present disclosure can perform recognition processing on each frame of a video stream to obtain an initial image containing a human face; then, the facial region in the initial image is aligned with a preset skin template to obtain a mapping relationship between the facial region and the preset skin template; the skin region of the facial region is adjusted to obtain an adjusted image; the skin region is determined based on the mapping relationship; then, the adjusted image is subjected to image quality enhancement processing to obtain an enhanced image; finally, the enhanced image is rendered to obtain a target image. In this way, through the facial adjustment processing and image quality enhancement processing in this embodiment, the quality of the target image can be improved, which is conducive to improving the user experience.

[0204] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0205] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0206] Fig. 1 is a flowchart showing an image processing method according to an exemplary embodiment.

[0207] Fig. 2 is a flow chart showing a method of acquiring a mapping relationship according to an exemplary embodiment.

[0208] Fig. 3 is a schematic diagram showing a preset skin template according to an exemplary embodiment.

[0209] Fig. 4 is a flow chart showing a method of acquiring a mapping relationship according to an exemplary embodiment.

[0210] Fig. 5 is a flow chart showing a method of acquiring an adjusted image according to an exemplary embodiment.

[0211] Fig. 6 is a flowchart showing a method of acquiring and processing an image according to an exemplary embodiment.

[0212] Fig. 7 is a flow chart showing a method of obtaining a spot area according to an exemplary embodiment.

[0213] Fig. 8 is a flowchart showing a method of acquiring a first candidate image according to an exemplary embodiment.

[0214] Fig. 9 is a flowchart showing a method of acquiring a second candidate image according to an exemplary embodiment.

[0215] Fig. 10 is a schematic diagram showing an elliptical area for enlarging an eye according to an exemplary embodiment.

[0216] FIG11 is a schematic diagram showing a circular area for face slimming according to an exemplary embodiment.

[0217] Fig. 12 is a flowchart showing a method of acquiring a third candidate image according to an exemplary embodiment.

[0218] Fig. 13 is a flow chart showing a method of acquiring an enhanced image according to an exemplary embodiment.

[0219] Fig. 14 is a schematic diagram showing a filter kernel according to an exemplary embodiment.

[0220] Fig. 15 is a flowchart showing a method for rendering an enhanced image according to an exemplary embodiment.

[0221] Fig. 16 is a schematic diagram showing data processing of a rendering pipeline according to an exemplary embodiment.

[0222] Fig. 17 is a flowchart showing an image processing method according to an exemplary embodiment.

[0223] Fig. 18 is a block diagram showing an image processing apparatus according to an exemplary embodiment. DETAILED DESCRIPTION

[0224] Exemplary embodiments will be described in detail herein, with examples shown in the accompanying drawings. When the following description refers to the drawings, identical numbers in different drawings represent identical or similar elements, unless otherwise indicated. The exemplary embodiments described below do not represent all embodiments consistent with the present disclosure. Rather, they are merely examples of devices consistent with certain aspects of the present disclosure, as detailed in the appended claims. It should be noted that, unless there is a conflict, the features of the following embodiments and implementations may be combined with each other.

[0225] To solve the above technical problems, the embodiments of the present disclosure provide an image processing method and apparatus, an electronic device, and a storage medium, which can be applied to a video conferencing system. The video conferencing system may include a server and multiple electronic devices, and its operating principle is as follows:

[0226] In one example, during a video conference, each participating electronic device can capture video and upload it to a server as a video stream. The server can then distribute the video stream to other participating electronic devices. At this point, the electronic device can capture each frame of video in the video stream and execute an image processing method provided by the present disclosure to obtain a target image corresponding to each frame. The target image can then be displayed, achieving a real-time beautification effect.

[0227] In one example, during a video conference, each participating electronic device can capture video and upload it to a server as a video stream. The server can then obtain each frame of video image in the video stream and execute an image processing method provided by the present disclosure to obtain a target image corresponding to each frame of video image. The server then forms a new video stream based on the sequence of the video frames and distributes it to each electronic device. The electronic device can then display the received target image, achieving a real-time beautification effect.

[0228] For ease of understanding, in each embodiment of the present disclosure, an electronic device in a video conferencing system is taken as an example to describe an image processing method provided by an embodiment of the present disclosure. FIG1 is a flowchart of an image processing method according to an exemplary embodiment.

[0229] Referring to FIG. 1 , an image processing method includes steps 11 to 15 .

[0230] In step 11, recognition processing is performed on each frame of the video stream to obtain an initial image containing a human face.

[0231] In this embodiment, the electronic device may obtain a video stream uploaded by an electronic device at the opposite end of the video conferencing system, where the video stream includes several frames of video images.

[0232] In this embodiment, a face recognition model is stored in the electronic device, and the face recognition model may include at least one of the following: a neural network model, an eigenface model, principal component analysis (PCA), or a support vector machine (SVM). In one example, the face recognition model is implemented using a neural network model. In another example, the face recognition model can be implemented using a facial feature recognition library, such as seetaface or dlib. Technicians can select a face recognition model based on the specific scenario, and the corresponding solution falls within the scope of protection of this disclosure.

[0233] In this embodiment, the facial recognition model can be pre-trained. The processor of the electronic device can sequentially input each frame of video image into the facial recognition model in chronological order and obtain a feature point vector output by the facial recognition model. The feature point vector includes the coordinate data and feature point numbers of a preset number of feature points. In one example, the preset number is 68, and the feature points are numbered 0-67, meaning that the feature point vector includes the coordinate data of 68 feature points.

[0234] In one example, when the feature point vector is null, it indicates that no face was detected in the video image. In another example, when the feature point vector is not null, it indicates that a face was detected in the video image. Thus, in this embodiment, face recognition processing can obtain a video image containing a face, which will be referred to as the initial image for ease of description.

[0235] In step 12, the face region in the initial image and the preset skin template are aligned to obtain a mapping relationship between the face region and the preset skin template.

[0236] In this step, the processor may align the face region in the initial image with the preset skin template to obtain a mapping relationship between the face region and the preset skin template, as shown in FIG2 , which includes steps 21 to 23 .

[0237] In step 21, the key point coordinates of the face in the preset skin template are obtained to obtain a first key point coordinate vector.

[0238] In this step, the electronic device stores a preset skin template, as shown in Figure 3. This preset skin template is in a teardrop or elliptical shape and is used to match the skin area of ​​a human face. The preset skin template can be configured with a preset number of feature points, which is the same as the number of feature points identified by the facial recognition model, to facilitate subsequent matching.

[0239] In this step, the processor can obtain the feature point vector of the preset skin template. The details are similar to the method for obtaining the facial feature points in step 11, and will not be repeated here. In this way, the processor can obtain the coordinates of the key points of the face in the preset skin template, and the preset number of key point coordinates can be used to generate a first key point coordinate vector. Key points can include eyebrows, eyes, nose tip and the triangle area within the facial region, upper and lower lips, cheeks, and jaw, etc., and can be set according to the specific scenario and are not limited here.

[0240] In step 22, the coordinates of key points of the face in the initial image are obtained to obtain a second key point coordinate vector.

[0241] In this step, the processor may obtain the coordinates of key points of the face in the initial image to obtain a second key point coordinate vector. For details, please refer to the solution of step 11 and will not be repeated here.

[0242] In step 23, the first key point coordinate vector and the second key point coordinate vector are input into a preset distance model and a solution of the preset distance model is obtained, and the solution of the preset distance model is used as the mapping relationship.

[0243] In this step, a preset distance model is stored in the electronic device, and the preset distance model is used to obtain the cumulative value of the square of the distance between the face area of ​​the initial image and the key point at the same position of the preset skin model. The preset distance model can be as shown in formula (1).

[0244] In formula (1), D represents the accumulated value of the square of the distance; i represents the ordinal number of the key point, which ranges from 0 to 67; s is a scalar, which represents the degree of geometric scaling; R is a 2*2 matrix, which represents the geometric rotation program; T is a two-dimensional vector, which represents the degree of geometric offset; pi represents the coordinate vector of the key point in the preset skin model, i.e., the first key point coordinate vector; qi represents the coordinate vector of the key point in the face area, i.e., the second key point coordinate vector; ||*|| 2 Indicates that the two key point vectors are subtracted and then the square sum is calculated; This indicates calculating the cumulative sum of the squares of the 68 distances from 0 to 67.

[0245] In this step, in order to ensure that the face area can better match the preset distance model, it is necessary to ensure that the preset distance model takes the minimum value. The processor obtains the solution of the preset distance model, see Figure 4, including steps 41 to 45.

[0246] In step 41, the processor obtains the coordinate mean and coordinate standard deviation of the first key point coordinate vector respectively to obtain a first coordinate mean and a first coordinate standard deviation.

[0247] In step 1, the processor may obtain the sum of the horizontal coordinates of each key point in the first key point coordinate vector, and then calculate the average horizontal coordinate value based on the sum and the number. The processor may obtain the sum of the vertical coordinates of each key point in the first key point coordinate vector, and then calculate the average vertical coordinate value based on the sum and the number. The average horizontal coordinate value and the average vertical coordinate value constitute the coordinate mean of the first key point coordinate vector, which is hereinafter referred to as the first coordinate mean and is denoted by c1.

[0248] In step 5, the processor may obtain the coordinate standard deviation of each key point in the first key point coordinate vector, which is subsequently referred to as the first coordinate standard deviation, as shown in formula (2).

[0249] In formula (2), s1 represents the standard deviation of the first coordinate, xi represents the horizontal coordinate of each key point, and c1 represents the mean of the first coordinate.

[0250] In step 42, the coordinate mean and the coordinate standard deviation of the second key point coordinate vector are respectively obtained to obtain a second coordinate mean and a second coordinate standard deviation.

[0251] In this step, the processor can obtain the coordinate mean and coordinate standard deviation of the second key point coordinate vector respectively to obtain the second coordinate mean c2 and the second coordinate standard deviation s2. The method for obtaining c2 and s2 can refer to the solution of step 41 and will not be repeated here.

[0252] In step 43 , the first coordinate mean is subtracted from the first key point coordinate vector and the resultant value is divided by the first coordinate standard deviation to obtain a first adjusted coordinate vector.

[0253] In this step, the processor may subtract the first coordinate mean from the first key point coordinate vector and divide the result by the first coordinate standard deviation to obtain a first adjusted coordinate vector, as shown in formula (3).

[0254] In formula (3), q1 represents the first adjustment coordinate vector.

[0255] In step 44 , the second coordinate mean is subtracted from the second key point coordinate vector and the resultant value is divided by the second coordinate standard deviation to obtain a second adjusted coordinate vector.

[0256] In this step, the processor may subtract the second coordinate mean from the second key point coordinate vector and divide the result by the second coordinate standard deviation to obtain a second adjusted coordinate vector, as shown in formula (4).

[0257] In formula (4), p1 represents the second adjustment coordinate vector.

[0258] In step 45 , the first adjusted coordinate vector and the second adjusted coordinate vector are input into the preset distance model and singular value decomposition is performed on the preset distance model to obtain the mapping relationship.

[0259] In this step, the processor may input the first adjustment coordinate vector and the second adjustment coordinate vector into the preset distance model, i.e., equation (1), to obtain equation (5).

[0260] At this time, the processor can perform singular value decomposition on the preset distance model to obtain the R matrix in formula (5), as shown in formula (6). U, S, V = SVD (p1 T q1); (6)

[0261] In formula (6), the matrix U represents the column vectors, which is the new orthonormal basis; the matrix V represents the column vectors, which is the orthonormal basis of the original domain; the matrix S represents the diagonal matrix, which represents the proportional (scaling) relationship between the vectors in the matrix V and the corresponding vectors in the matrix U.

[0262] The relationship between matrix R and matrices U and V is shown in formula (7). R = (U*V) T ;(7)

[0263] Then, the processor can obtain the mapping relationship M, as shown in formula (8).

[0264] In formula (8),

[0265] That is to say, the processor can align the 68 key points of the preset skin template with the key points of the face area after the alignment transformation of the mapping relationship M, so as to achieve the effect of aligning the face area and the preset skin template in the geometric space.

[0266] Finally, the processor can obtain the coordinate data of the pixel points of the preset skin model according to the coordinate data of the pixel points of the face area and the above mapping relationship, as shown in formula (9).

[0267] In formula (9), x * and y * x and y represent the horizontal and vertical coordinates of the pixel points in the face area.

[0268] In step 13, the skin area of ​​the face area is adjusted to obtain an adjusted image; the skin area is determined according to the mapping relationship.

[0269] In this embodiment, the processor may perform face adjustment processing on the face region in the initial image to obtain an adjusted image, as shown in FIG5 , which includes steps 51 and 52 .

[0270] In step 51 , in response to sequentially scaling the initial image to obtain candidate size images of preset size ratios, the candidate size images are cleared to obtain processed images.

[0271] In this embodiment, the electronic device stores preset size ratios. These preset size ratios refer to the ratios by which the initial image is magnified or reduced, and can be set based on specific scenarios. In one example, the preset size ratios refer to magnification ratios, and a preset number of magnification ratios can be set, such as 2x, 4x, 8x, etc. Thus, the processor can sequentially magnify the initial image to 2x, 4x, 8x, etc., according to the preset size ratios.

[0272] In this embodiment, the processor may sequentially scale the initial image to the aforementioned preset size ratios to obtain a candidate size image. The electronic device may then perform a cleanup process on the candidate size image to remove abnormal regions in the candidate size image, including spot regions and / or stripe regions.

[0273] In one embodiment, the processor may perform a clearing process on the candidate size image to obtain a processed image, as shown in FIG6 , which includes steps 61 to 63 .

[0274] In step 61 , the processor may detect at least one spot region in the skin region of the candidate size image, as shown in FIG. 7 , which includes steps 71 to 73 .

[0275] In step 71, the processor may obtain target pixel data according to the mapping relationship and the pixel data of each pixel point of the preset skin template. The obtaining method is as shown in formula (9), which will not be repeated here.

[0276] In step 72 , the processor may replace the pixel data at the corresponding position in the candidate size image with the target pixel data to obtain a second candidate image.

[0277] In step 73, the processor may perform connected domain detection on the skin area of ​​the second candidate image to obtain a spot area of ​​the skin area, wherein a connected domain whose area is within a first preset area range and whose size is smaller than a first preset size threshold and larger than a second preset size threshold is used as the spot area of ​​the skin area.

[0278] In this step, the processor may obtain a grayscale image corresponding to the candidate size image. For details, please refer to the solution for converting an RGB image into a YUV image, which will not be described in detail here.

[0279] The processor can convert the grayscale image into a binary image according to each preset threshold value to obtain multiple frames of binary images. Wherein, the value range of the preset threshold value is [T1, T2], the step size is t, and the preset threshold values ​​are T1, T1+t, T1+2t, T1+3t, ... and T2 in sequence.

[0280] Then, the processor can obtain the connected areas of each binary image frame respectively to obtain multiple connected domains of each binary image frame. The connected areas refer to the closed areas obtained when detecting the boundaries of each binary image, and each of the above connected areas is a spot of the binary image.

[0281] The processor may then classify the multiple connected domains in the multiple binary image frames based on the distances between the center points of the connected domains to obtain multiple connected domains whose center point distances are less than a preset distance threshold. If the distances between the center points of connected domains in the multiple binary image frames are less than the preset distance threshold, it indicates that the multiple binary image frames contain the same connected domain. There is a high probability that the connected domain is a speckle, and further, it is likely that the connected domain is a speckle region in the grayscale image.

[0282] In other words, in this example, the processor may cluster the connected domains of the multiple two-frame images and classify the connected domains according to each cluster group.

[0283] Furthermore, the processor may obtain, as the spot region of the skin region, a connected region whose position and size are less than a first preset size threshold and greater than a second preset size threshold. The position refers to the average coordinates of the center points of the connected regions in each binary image frame; the size refers to the radius of a circumscribed circle when the area of ​​the connected region is within a preset area range. The first preset area range is [6, 100] pixels.

[0284] Finally, the processor can identify connected domains whose size is smaller than a first preset size threshold and larger than a second preset size threshold as spot regions in the skin region, thereby obtaining at least one spot region. The preset sizes range from (1, 5). The first preset size threshold is 5, and the second preset size is [1, 3]. Thus, this step can include connected domains that are either too small or too large.

[0285] It should be noted that after the spot area is determined, the spot area can be marked on the preset skin template to mark the position and size of the spot area.

[0286] In step 62 , the processor may perform a clearing process on the at least one spot region to obtain a first processed image.

[0287] In this step, the processor can obtain pixel data surrounding the speckled area and use the pixel data to fill the pixels of the speckled area, for example, directly or after averaging, so that the pixels of the speckled area are the same or similar to the pixels of the surrounding neighbors, thereby eliminating the speckled area. In this way, the processor can obtain a first processed image.

[0288] In step 63, in response to satisfying the first loop condition, the processor can restore the first processed image to the size of the initial image to obtain the final processed image; the first loop condition means that the number of times the spot area is detected is equal to the first preset number threshold or the number of the spot areas is less than the first preset number threshold.

[0289] In this step, the processor can determine whether a first loop condition is currently satisfied. The first loop condition is that the number of times the spot area is detected is equal to a first preset number threshold or the number of the spot areas is less than the first preset number threshold. The first preset number threshold has a value range of [1, 10]. In one example, the first preset number threshold has a value of 3. The first preset number threshold has a value range of [1, 10]. In one example, the first preset number threshold has a value of 2.

[0290] Taking the example of the first loop condition being that the number of times a speckled area is detected is equal to a first preset threshold, the processor may continuously execute steps 61 and 62 and record the number of executions. Since each execution of step 61 requires detecting a speckled area once, the number of executions is equal to the number of times the speckled area is detected. If the number of times a speckled area is detected is less than the first preset threshold, the first loop condition is determined to be unsatisfied, and step 61 is continued until the number of times a speckled area is detected is less than the first preset threshold, determining that the first loop condition is satisfied and stopping execution of step 61.

[0291] Taking the first loop condition that the number of spot areas is less than the first preset number threshold as an example, the processor can continue to execute steps 61 and 62, and record the number of spot areas detected each time. When the number of times the spot areas are detected is greater than or equal to the first preset number threshold, it is determined that the first loop condition is not met, and step 61 is continued to be executed until the number of times the spot areas are detected is less than the first preset number threshold, it is determined that the first loop condition is met and step 61 is stopped.

[0292] In this step, by setting the first loop condition, the skin area of ​​the same candidate size image can be detected multiple times, which can avoid missing the spot area and ensure the effect of removing the spot area.

[0293] In one embodiment, the processor can also determine whether a stripe region exists when no speckle region is detected. The scheme for the stripe region is similar to that for the speckle region, except that the stripe region and the speckle region have different areas and sizes. The stripe region, which is different from the speckle region, can be selected by adjusting the screening condition for the connected domain, i.e., the subsequent second preset condition.

[0294] For example, the second preset condition may include: the second preset area range is [6, 200] pixels, the preset length range is [5, 20] pixels, and the preset width range is [1, 5] pixels. It can be set according to the specific scenario and is not limited here.

[0295] When at least one striped region is detected, the processor may remove the striped regions that meet the second preset condition in the candidate size image to obtain a second processed image. The striped region removal process is the same as the speckle removal process, and the speckle removal process is illustrated in FIG6 , which is not further described here.

[0296] The processor can determine whether a second loop condition is satisfied, where the number of striped regions detected is equal to a second preset number threshold or the number of striped regions is less than the second preset number threshold. The second preset number threshold can be set in a manner similar to the first preset number threshold. The second preset number threshold can be set in a manner similar to the second preset number threshold. In response to satisfying the second loop condition, the processor can restore the second processed image to the size of the original image, thereby obtaining a final processed image, i.e., an adjusted image.

[0297] In one embodiment, when the abnormal area includes a stripe area, the processor may detect the stripe area of ​​the skin area of ​​the candidate size image; then, in response to detecting at least one stripe area, the processor may clear the stripe area in the candidate size image that meets the second preset condition to obtain a third processed image; then, in response to meeting the second loop condition, the third processed image is restored to the size of the initial image to obtain a final processed image; the second loop condition refers to the number of times the stripe area is detected is equal to the second preset number threshold or the number of the stripe areas is less than the second preset number threshold.

[0298] In another embodiment, the processor may first detect the stripe area for clearing processing, and then detect the spot area for clearing processing, wherein the detection scheme and the clearing processing scheme are the same as those of the above embodiments, the difference being that the order of detecting the stripe area and the spot area is changed, and the effect of clearing processing abnormal areas can also be achieved.

[0299] In step 52 , in response to the number of resizing operations being equal to a preset threshold, the final processed image is restored to the size of the initial image to obtain a first candidate image, which is used as the adjusted image.

[0300] In this embodiment, each time the size of the initial image is adjusted, the processor may record the number of resizing operations and determine the difference between the number of resizing operations and a preset number threshold. When the number of resizing operations equals the preset number threshold, the processor may restore the final processed image (with the abnormal area removed) to the size of the initial image, thereby obtaining a first candidate image. In some embodiments, the processor may use this first candidate image as the adjusted image.

[0301] It is understandable that the embodiment shown in FIG2 discloses a solution for obtaining abnormal regions of each candidate size image and clearing and processing the abnormal regions of the candidate size image, that is, a solution for clearing and processing abnormal regions of each candidate size image.

[0302] In practical applications, the processor can obtain abnormal regions and mark the locations of each candidate size image after each acquisition. After a preset number of zooms, the processor can then clear the abnormal regions of the final candidate size image based on the abnormal region markings of the preset number of candidate size images. This means that the processor obtains abnormal regions from multiple candidate size images and only clears all abnormal regions on the final image. Referring to FIG8 , the processor can perform clearing processing on the candidate size images to obtain a processed image, including steps 81 to 83.

[0303] In step 81, the processor may align the face region in the initial image with the preset skin template to obtain a mapping relationship between the face region and the preset skin template.

[0304] In this step, the acquisition scheme of the above mapping relationship is the same as that of step 33. For details, please refer to the content of step 33 and will not be repeated here.

[0305] In step 82, the processor can scale the initial image into candidate size images of different preset sizes in sequence, obtain abnormal areas of the skin area in the candidate size images, and obtain an abnormal area feature map; the abnormal areas include spot areas and / or stripe areas.

[0306] In this step, the processor may sequentially scale the initial image into candidate size images of different preset sizes. For example, the electronic device may store preset size ratios, which refer to the ratio by which the initial image is magnified or reduced, and may be set based on the specific scenario. In one example, the preset size ratio refers to a magnification ratio and may be set to a preset number, such as 2x, 4x, 8x, etc. Thus, the processor may sequentially magnify the initial image to 2x, 4x, 8x, etc., of its original size according to the preset size ratios.

[0307] In this step, in response to the initial image being scaled sequentially into candidate-sized images of the same preset size, the processor can obtain abnormal skin regions within the candidate-sized images. When the abnormal region is a spot region, the method for obtaining the spot region is the same as that illustrated in FIG7 . For details, see FIG7 and will not be repeated here. When the abnormal region is a stripe region, the method for obtaining the stripe region is the same as that illustrated in FIG2 and will not be repeated here.

[0308] In this step, after obtaining the abnormal area of ​​each candidate size image, the abnormal area can be further identified on the preset skin template, that is, the position and size of the abnormal area are identified, so that the abnormal areas of a preset number of candidate size images can be summarized.

[0309] In step 83 , in response to the number of resizing operations being equal to a preset threshold, the processor may clear the abnormal processing area on the last candidate size image to obtain the adjusted image.

[0310] In this step, when the number of resizing operations reaches a preset threshold, the processor removes the abnormal regions detected in step 82 based on the last candidate size image. In other words, abnormal regions are detected in candidate size images of different sizes, and all detected abnormal regions are removed from the last candidate size image to obtain the first candidate image. For example, if there are three candidate size images, named Candidate 1, Candidate 2, and Candidate 3, and the number of abnormal regions in each is 5, 6, and 7, the processor needs to remove (5 + 6 + 7 =) 18 abnormal regions in the candidate size image named Candidate 3.

[0311] It should be noted that when the scaling ratio is set reasonably and the filtering conditions are set reasonably, the abnormal areas of each candidate size image can achieve the effect of not overlapping with each other. For example, the abnormal area in candidate 1 cannot become the abnormal area in candidate 2 because its area is too small, and the abnormal area in candidate 2 cannot become the abnormal area in candidate 3 because its area is too small, which is beneficial to improve the effect of clearing abnormal areas in the skin area.

[0312] In one embodiment, the first candidate image obtained in step 23 and step 83 can be used as the adjustment image, and in this case, the process can jump to step 13 .

[0313] In another embodiment, the processor may perform scaling processing on the eye region in the first candidate image to obtain a second candidate image, as shown in FIG. 9 , which includes steps 91 to 93 .

[0314] In step 91 , the processor obtains a scaling ratio of each pixel in the eye region of the first candidate image.

[0315] Considering the shape of the human eye is close to an ellipse, this embodiment creates an elliptical region around the center point C of the eye and improves the magnification of this elliptical region. Referring to Figure 10 , for any point P(x,y) within the elliptical region, a new position is selected from the line CP passing through the center point C(x0,y0) and P(x,y). If the new position is between the two points CP, the new position is P1, which reduces the size of the eye. If it is on the extension of the line CP, the new position is P2, which increases the size of the eye.

[0316] The processor can obtain the distance ratio m between the distance between the point P(x, y) and the center point C(x0, y0) and the distance ratio of the ellipse radius combination, that is, the processor can obtain the distance ratio of each pixel point in the eye area of ​​the first candidate image. The distance ratio is used to indicate that the closer the pixel point is to the center point of the eye area, the smaller the moving distance ratio is, as shown in formula (10).

[0317] In formula (10), m represents the distance ratio, which means that the closer the pixel is to the center point C, the smaller the scaling ratio is, and the farther the pixel is from the center point C, the larger the scaling ratio is; x and y represent the horizontal and vertical coordinates of the pixel point P, respectively; x0 and y0 represent the horizontal and vertical coordinates of the center point C, respectively; a represents the major axis of the ellipse, a = 2*distance (left corner of the eye, right corner of the eye); b represents the minor axis of the ellipse, b = 2*distance (upper eyelid, lower eyelid); and distance is a function for calculating the distance between two points.

[0318] The coefficient 2 is set before calculating the distance function of the values ​​of a and b, indicating that the eye area is magnified, so as to achieve the effect of synchronously scaling the area around the eye, thereby ensuring a more natural eye scaling effect.

[0319] Then, the processor can obtain the scaling ratio of each pixel point according to the distance ratio, where the scaling ratio is proportional to the distance ratio, as shown in formula (11). K = 1-K1*(1-m) 2 ;(11)

[0320] In formula (11), K represents the scaling ratio, K1 represents the first initial value of the scaling ratio, and m represents the distance ratio.

[0321] In step 92, target coordinate data of each pixel point is acquired according to the scaling ratio and the original coordinate data of each pixel point.

[0322] In this step, the processor can obtain the target coordinate data of each pixel point according to the scaling ratio and the original coordinate data of each pixel point, as shown in formula (12). x'=x0+K(x-x0) y'=y0+K(y-y0); (12)

[0323] In formula (12), x, y represent the horizontal coordinate and vertical coordinate of the original coordinate data of each pixel point respectively; x', y' represent the horizontal coordinate and vertical coordinate of the target coordinate data of each pixel point respectively; x0, y0 represent the horizontal coordinate and vertical coordinate of the center point C respectively.

[0324] In step 93, the original coordinate data of each pixel in the first candidate image is replaced with the target coordinate data to obtain the second candidate image.

[0325] In this step, the processor may replace the original coordinate data (x, y) of each pixel point in the first candidate image with the target coordinate data (x', y') to obtain the second candidate image.

[0326] In this way, in this embodiment, the eye region in the first candidate image is enlarged to achieve the effect of enlarging the eyes.

[0327] In one embodiment, the second candidate image may be used as the adjustment image. In this case, the processor may jump to step 13 .

[0328] In one embodiment, the processor can perform face slimming on the user. Referring to FIG11 , a control point C1 is set in the facial region. All pixels within a circular region with a radius of Rmax and a center point C1 are moved. N is a target point in the facial region. When control point C1 is moved toward target point N, all pixels within the circular region are moved.

[0329] Point X is any point within the circular region. When control point C1 moves to target point N, the pixel data at point X should be moved from point U. Thus, after determining the coordinates of points C1, N, and X, as well as the radius length Rmax, the coordinate data of point U can be calculated. The pixel data at point U is then replaced with the pixel data at point X, achieving the image deformation effect.

[0330] In this embodiment, the processor may resize the face region in the second candidate image to obtain a third candidate image, as shown in FIG. 12 , which includes steps 121 to 124 .

[0331] In step 121, the processor may obtain the distance value between the control point C1 of the face region and each pixel point in the face region. For example, the distance between point X and point C1 is d=distance(C1,X).

[0332] In step 122, the processor may obtain the deformation ratio of each pixel point according to the distance value, wherein the deformation ratio is used to indicate that the farther the pixel point is from the control point C1, the smaller the movement distance. In this step, the deformation ratio of each pixel point is shown in formula (13).

[0333] In formula (13), f represents the deformation ratio. The farther away the pixel is from the center point C1, the smaller the deformation is, and the closer the pixel is to the center point C1, the larger the deformation is. d represents the distance between the pixel point X and the control point C1. Rmax represents the radius of the deformation area around the control point C1.

[0334] In step 123, the processor may calculate the target pixel data of each pixel point based on the coordinate data of the target position N of the control point C1, the coordinate data of the control point C1, and the coordinate data of the target position X of each pixel point. In this step, the processor obtains the target pixel data as shown in formula (14).

[0335] In formula (14), U represents the target pixel data, X represents the pixel data of point X, N represents the pixel data of point N, C1 represents the pixel data of the control point, f represents the deformation ratio, and K2 represents the second initial value of the scaling ratio.

[0336] In step 124 , the processor may replace the pixel points in the face area of ​​the second candidate image with the target pixel data to obtain the third candidate image.

[0337] In this step, the processor may use the third candidate image as the adjustment image.

[0338] It should be noted that this embodiment describes a scheme for clearing abnormal areas, enlarging eyes, and thinning the face. In some possible embodiments, the order of clearing abnormal areas, enlarging eyes, and thinning the face can be adjusted. For example, a combination of clearing abnormal areas, thinning the face, and enlarging eyes; a combination of thinning the face, enlarging eyes, and clearing abnormal areas; a combination of enlarging eyes, thinning the face, and clearing abnormal areas; a combination of enlarging eyes, clearing abnormal areas, and thinning the face; a combination of thinning the face, clearing abnormal areas, and enlarging eyes; and other various combinations. In the absence of any conflict, the resulting scheme falls within the scope of protection of this disclosure.

[0339] In step 14, the adjusted image is subjected to image quality enhancement processing to obtain an enhanced image.

[0340] In this embodiment, the processor may perform image quality enhancement processing on the adjusted image to obtain an enhanced image, as shown in FIG13 , which includes steps 131 to 135 .

[0341] In step 131 , the processor may perform skin resurfacing processing on the adjusted image to obtain a first skin resurfacing feature map.

[0342] In this step, the processor can filter each pixel of the green sub-image of the adjusted image to obtain a filtered sub-image. The filtering method used in the filtering process includes one of the following: averaging filtering, Gaussian filtering, bilateral filtering, and guided filtering. In one example, the filtering process is implemented using Gaussian filtering, and the filter kernel of the Gaussian filter is implemented using a circular filter kernel, and the sum of the weight values ​​of the circular filter kernel is equal to the preset weight value. In one example, the preset weight value is 62, and the circular filter kernel is shown in Figure 14.

[0343] Then, the processor may obtain the difference in pixel data of the same pixel point in the adjusted image and the filtered sub-image to obtain a difference sub-image.

[0344] Afterwards, the processor can compensate the pixel data of each pixel in the difference sub-image based on the preset compensation data to obtain the difference-compensated sub-image. Taking the preset compensation data as 0.5 as an example, the difference-compensated sub-image = green sub-image - filtered pixel value + 0.5.

[0345] It should be noted that the grayscale value of the green sub-image after filtering is less than 0, while the grayscale value of normal skin is positive. Therefore, a preset compensation data is added for normal compensation. After compensation, the pixel data of the pixel point of the difference compensation sub-image is less than 0.5, while the value of normal skin is greater than 0.5.

[0346] Finally, the processor may perform light compensation on the difference-compensated sub-image to obtain the first skin resurfacing feature map.

[0347] Considering that the difference-compensated sub-image is grayish, light compensation is performed on the difference-compensated sub-image in this step, so that the features of the difference-compensated sub-image are more obvious. In one example, the processor performs light compensation on the difference-compensated sub-image at least once. For example, the processor can compare the pixel data of each pixel of the difference-compensated sub-image with the preset compensation data; when the pixel data of each pixel is greater than or equal to the preset compensation data (such as 0.5), the processor can obtain the target pixel data corresponding to the pixel data according to the first light compensation model; when the pixel data of each pixel is less than the preset compensation data, the processor can obtain the target pixel data corresponding to the pixel data according to the second light compensation model; then, the processor can replace the pixel data of each pixel of the difference-compensated sub-image with the target pixel data. Finally, in response to the number of compensations reaching the preset compensation number threshold, the processor can determine that the difference-compensated sub-image obtained by the last compensation is the first skin resurfacing feature map. In other words, the processor performs light compensation on the difference-compensated sub-image as shown in formula (15).

[0348] In formula (15), x represents the input value, y represents the output value, and y = 2*x 2 Represents the first light compensation model, y = 1-2*(1-x) 2 Represents the second light compensation model.

[0349] In one example, the preset compensation times threshold is in the range of [1, 10]. After the processor compensates for the light, it can obtain a first skin resurfacing feature map, which is recorded as Highpass.

[0350] In step 132, the processor may adjust the skin resurfacing degree of the skin resurfacing feature map to obtain a second skin resurfacing feature map.

[0351] In this step, the processor can obtain the grayscale image corresponding to the adjusted image, as shown in formula (16). gray = 0.299*r + 0.587*g + 0.114*b; (16)

[0352] In formula (16), r, g, and b represent the red, green, and blue pixel data of the adjusted image, respectively, and gray represents the grayscale value of the pixel point of the grayscale image.

[0353] Then, the processor can perform a brightening process on the grayscale data of each pixel in the grayscale image based on the preset skin resurfacing degree data to obtain a grayscale brightened image, as shown in formula (17). mgray=gray 1-0.6*K3 ; (17)

[0354] In formula (17), K3 represents the degree of skin resurfacing set by the user, and its value range is [0, 1]. The larger K3 is, the higher the degree of skin resurfacing is.

[0355] Afterwards, the processor may adjust the pixel data of each pixel point of the adjusted image according to the first skin resurfacing feature map and the grayscale brightened image to obtain the second skin resurfacing feature map, as shown in formula (18).

[0356] In formula (18), Respectively represent the pixel data of the pixel points of the first skin resurfacing feature map, Respectively represent pixel data of pixel points of the second skin resurfacing feature map.

[0357] In step 133, the processor may perform skin lightening processing on the second skin resurfacing feature map to obtain a skin lightening feature map.

[0358] In this step, the processor can adjust the pixel data after skin refining of the second skin refining feature map, as shown in formula (19).

[0359] In formula (19), Represents pixel data after skin smoothing adjustment.

[0360] Then, the processor can perform skin lightening processing on the second skin resurfacing feature map, as shown in formula (20).

[0361] In formula (20), Pixel data representing a pixel of the skin lightening feature map.

[0362] In this step, the original pixel data after skin smoothing is superimposed on the original pixel data through the screen color filter mode, so that the standard color values ​​of the upper and lower pixels are inverted and then multiplied and output. The output result is brighter than the pixel data of the two, achieving the effect of brightening the skin tone.

[0363] In step 134 , the processor may perform soft light processing based on the skin color brightening feature map and the adjusted image to obtain a soft light feature map.

[0364] In this step, the processor obtains the soft light feature map as shown in formula (21).

[0365] In formula (21), Pixel data representing a pixel point of the pre-light feature map.

[0366] In step 135 , the processor may obtain the enhanced image according to the adjusted image, the second skin resurfacing feature map, the skin tone brightening feature map, the soft light feature map, and their respective weight values.

[0367] In this step, the processor obtains the enhanced image as shown in formula (22).

[0368] In formula (22), Represents the pixel data of the enhanced image, a+b+c=1, and bright is a preset brightening parameter.

[0369] In step 15, the enhanced image is rendered to obtain a target image.

[0370] In this embodiment, an OpenGL (Open Graphics Library) object is pre-deployed within the electronic device. By invoking OpenGL instructions and calling the API provided by OpenGL, rendering operations are performed on the enhanced image, which is ultimately displayed on the electronic device's display. This allows for direct rendering and display, ensuring real-time performance in video conferencing scenarios.

[0371] In one embodiment, OpenGL ES 3.0 is deployed in the electronic device, and its rendering process is shown in FIG15 .

[0372] (1) Define the Vertex Shader. Use the in keyword to declare all input vertex attributes in the vertex shader. The vertex shader disclosed in this disclosure is relatively simple and only needs to declare two input vertex attributes: the input vertex coordinates Vertex and the input texture coordinates Texcord. Use the out keyword to declare all output vertex attributes and the output texture coordinates Texcord in the vertex shader.

[0373] (2) Define the fragment shader. Use the in keyword to declare the texture coordinate input attributes in the fragment shader, and the uniform keyword to declare the global variable attributes in the fragment shader. The input global variables of the fragment shader are the image object sample2D image, the parameter array paras, and the texture coordinates output by the vertex shader. Use the out keyword to declare all output attributes in the fragment shader, and output the color value OutColor corresponding to the texture coordinate.

[0374] (3) Define the shader program and attach the vertex shader and fragment shader to the shader program.

[0375] (4) Define the vertex array, texture array, and element array. Since the image is a square, these three arrays are fixed, and their definition values ​​are as follows:

[0376] (5) Define the vertex array object VAO1 (Vertex Array Object,) and bind and activate it.

[0377] (6) Define the vertex buffer object VBO1 (Vertex Buffer Objects), call the glBufferData function, and copy the previously defined vertex coordinate array Vertex array to the memory of the VBO1 buffer.

[0378] Define the vertex buffer object VBO2 (Vertex Buffer Objects), call the glBufferData function, and copy the previously defined texture coordinate array Texcord array into the memory of the VBO2 buffer.

[0379] Define the element buffer object EBO (Element Buffer Object) and copy the previously defined element index array Element array into the memory of the EBO buffer.

[0380] (7) Link VBO1 to the vertex attribute 0 (vertex coordinates) of the vertex shader and activate attribute 0; link VBO2 to the vertex attribute 1 (texture coordinates) of the vertex shader and activate attribute 1;

[0381] (8) Define the texture object Texture, input the video frame image, and copy the data into the memory of the texture object.

[0382] (9) Assign the texture Texture to the image object sample2D image of the fragment shader.

[0383] (10) Define various parameters for portrait beautification and assign them to the fragment shader parameter paras.

[0384] (11) Compile and link the shader program, activate VAO1, and bind FrameBuffer.

[0385] (12) Draw the image and render it to the screen.

[0386] In this embodiment, an abstract diagram of the data flow and processing steps in the rendering pipeline is shown in Figure 16. The input vertex data is first located by the vertex shader, and then the primitives are assembled according to the element buffer object, generally triangle assembly. The assembled primitives are rasterized into a grid, each grid corresponds to a pixel fragment, and then the fragment shader is used to render the color of each fragment. The color comes from the input image texture, and the image texture is processed by the input parameters. The output of the fragment shader undergoes testing and blending operations, also known as the Alpha test and blending stage. This stage detects the corresponding depth values ​​of the fragments and uses them to determine whether the pixel is in front of or behind other objects to decide whether it should be discarded. This stage also checks the alpha value (the alpha value defines the transparency of an object) and blends the objects. This stage uses the default settings and is ultimately rendered to the screen.

[0387] It should be noted that the shaders (Vertex Shader and Fragment Shader) in FIG15 are defined in the process of implementing the disclosed solution, and the processing of the gray part is processed by OpenGL by default and cannot be changed by the user.

[0388] Thus, the solution provided by the disclosed embodiment can perform recognition processing on each frame of a video stream to obtain an initial image containing a face; then, perform face adjustment processing on the facial region in the initial image to obtain an adjusted image; then, perform image quality enhancement processing on the adjusted image to obtain an enhanced image; and finally, render the enhanced image to obtain a target image. Thus, through face adjustment processing and image quality enhancement processing in this embodiment, the quality of the target image can be improved, which is conducive to improving the user experience.

[0389] The following describes an image processing method provided by an embodiment of the present disclosure in conjunction with an embodiment, referring to FIG17 , including:

[0390] After receiving the video stream, the electronic device performs facial recognition on each video image in the video stream to obtain an initial image containing a face, as described in step 11. Keypoint detection is then performed on the initial image to obtain a keypoint vector, which is then mapped to a preset skin template to obtain a mapping relationship, as described in the example of FIG3. Abnormal areas within the skin region, such as spots and / or stripes, are then detected based on the mapped keypoints and the detected abnormal areas are removed, as described in FIG6 or FIG8. The electronic device can then perform facial modification on the first candidate image after the abnormal areas have been removed, such as by slimming the face or enlarging the eyes, as described in FIG9 for enlarging the eyes and in FIG10 and FIG12 for slimming the face, thereby obtaining an adjusted image. Furthermore, the electronic device can enhance the image quality of the adjusted image, including, for example, skin smoothing, whitening, or brightening, as described in step 13, to obtain an enhanced image. Finally, the electronic device can render the enhanced image to obtain a target image, as described in step 14, and display the target image.

[0391] After receiving the video stream, the electronic device performs facial recognition on each video image in the video stream. If no face is detected, the video image is enhanced, as shown in step 13, to produce an enhanced image. Finally, the electronic device renders the enhanced image to the target image, as shown in step 15, and displays the target image.

[0392] According to a second aspect of an embodiment of the present disclosure, there is provided an image processing apparatus, as shown in FIG18 , comprising:

[0393] The initial image acquisition module 181 is used to perform recognition processing on each frame of the video stream to obtain an initial image containing a human face;

[0394] A mapping relationship acquisition module 182 is configured to align the face region in the initial image with a preset skin template to obtain a mapping relationship between the face region and the preset skin template;

[0395] An adjusted image acquisition module 183 is configured to adjust the skin area of ​​the face area to obtain an adjusted image; the skin area is determined according to the mapping relationship;

[0396] An enhanced image acquisition module 184 is configured to perform image quality enhancement processing on the adjusted image to obtain an enhanced image;

[0397] The target image acquisition module 185 is used to render the enhanced image to obtain a target image.

[0398] Optionally, the mapping relationship acquisition module includes:

[0399] A first vector acquisition submodule is used to obtain the key point coordinates of the face in the preset skin template to obtain a first key point coordinate vector;

[0400] A second vector acquisition submodule is used to obtain the key point coordinates of the face in the initial image to obtain a second key point coordinate vector;

[0401] A mapping relationship acquisition submodule is used to input the first key point coordinate vector and the second key point coordinate vector into a preset distance model and obtain a solution of the preset distance model, and use the solution of the preset distance model as the mapping relationship.

[0402] Optionally, the mapping relationship acquiring unit includes:

[0403] A first vector processing unit, configured to respectively obtain a coordinate mean and a coordinate standard deviation of the first key point coordinate vector to obtain a first coordinate mean and a first coordinate standard deviation;

[0404] A second vector processing unit, configured to respectively obtain a coordinate mean and a coordinate standard deviation of the second key point coordinate vector to obtain a second coordinate mean and a second coordinate standard deviation;

[0405] a first vector adjustment unit, configured to subtract the first coordinate mean from the first key point coordinate vector and divide the result by the first coordinate standard deviation to obtain a first adjusted coordinate vector;

[0406] a second vector adjustment unit, configured to subtract the second coordinate mean from the second key point coordinate vector and divide the resultant value by the second coordinate standard deviation to obtain a second adjusted coordinate vector;

[0407] A mapping relationship acquisition unit is configured to input the first adjusted coordinate vector and the second adjusted coordinate vector into the preset distance model and perform singular value decomposition on the preset distance model to obtain the mapping relationship.

[0408] Optionally, the adjustment image acquisition module includes:

[0409] a processed image acquisition submodule, configured to, in response to scaling the initial image in sequence to candidate size images of preset size ratios, perform clearing processing on the candidate size images to obtain processed images;

[0410] The first candidate image acquisition submodule is configured to restore the final processed image to the size of the initial image in response to the number of resizing operations being equal to a preset number threshold, to obtain a first candidate image, and use the first candidate image as the adjusted image.

[0411] Optionally, the processing image acquisition submodule includes:

[0412] a spot region detection unit, configured to detect at least one spot region in the skin region of the candidate size image; the spot region refers to a connected domain having an area within a first preset area range and a size smaller than a first preset size threshold and larger than a second preset size threshold;

[0413] a first processed image acquisition unit, configured to perform a clearing process on the at least one spot region to obtain a first processed image;

[0414] a processed image acquisition unit, configured to restore the first processed image to the size of the initial image in response to satisfying the first loop condition, to obtain a final processed image;

[0415] The first cycle condition means that the number of times the spot areas are detected is equal to a first preset number threshold or the number of the spot areas is less than the first preset number threshold.

[0416] Optionally, the spot area detection unit includes:

[0417] a target pixel acquisition subunit, configured to acquire target pixel data according to the mapping relationship and the pixel data of each pixel point of the preset skin template;

[0418] A second candidate image acquisition subunit is configured to replace pixel data at a corresponding position in the candidate size image with the target pixel data to obtain a second candidate image;

[0419] The spot area acquisition subunit is used to perform connected domain detection on the skin area of ​​the second candidate image, and obtain a connected domain whose spot area size of the skin area is smaller than a first preset size threshold and larger than a second preset size threshold as the spot area of ​​the skin area.

[0420] Optionally, the spot area acquisition subunit includes:

[0421] A binary image acquisition subunit is used to convert the grayscale image into a binary image according to each preset threshold value to obtain multiple frames of binary images;

[0422] A first connected domain acquisition subunit is used to respectively acquire connected areas of the binary image of each frame to obtain multiple connected domains of the binary image of each frame;

[0423] a second connected domain acquisition subunit, configured to classify the multiple connected domains according to the distances between the center points of the connected domains of the multiple binary image frames, and obtain multiple connected domains whose distances between the center points are less than a preset distance threshold;

[0424] The spot area acquisition subunit is used to obtain the positions and sizes of the multiple connected domains, and to use the connected domains whose sizes are smaller than a first preset size threshold and larger than a second preset size threshold as the spot areas of the skin area; the position refers to the average value of the center point coordinates of the connected domains of each frame of the binary image; the size refers to the radius length of the circumscribed circle when the area of ​​the connected domain of the binary image is within a preset area range.

[0425] Optionally, after detecting at least one spot area in the skin area of ​​the candidate size image, the processing image acquisition submodule further includes:

[0426] a stripe region determination unit, configured to determine whether a stripe region exists in response to no spot region being detected; wherein the stripe region refers to a connected domain having an area within a second preset area range, a size within a preset length range, and a width within a preset width range;

[0427] a spot area removing unit, configured to, in response to detecting at least one stripe area, remove the stripe area in the candidate size image that meets a second preset condition, to obtain a second processed image;

[0428] a processed image acquisition unit, configured to restore the second processed image to the size of the initial image in response to satisfying the second loop condition, to obtain a final processed image;

[0429] The second cycle condition means that the number of times the stripe regions are detected is equal to a second preset number threshold or the number of the stripe regions is less than a second preset number threshold.

[0430] Optionally, the first processed image acquisition unit includes:

[0431] a spot region acquisition subunit, configured to acquire at least one spot region meeting a preset condition;

[0432] The first processed image acquisition subunit is configured to perform a clearing process on the at least one spot subregion to obtain the first processed image.

[0433] Optionally, the spot area acquisition subunit includes:

[0434] a spot area acquisition subunit, configured to acquire the area of ​​each spot area within the skin area;

[0435] a candidate spot acquisition subunit, configured to acquire a spot region whose area is within a first preset area range to obtain a candidate spot region;

[0436] a spot region acquisition subunit, configured to acquire a candidate spot region whose roundness is equal to a preset roundness value and whose convexity is equal to a preset convexity value, to obtain the at least one spot region;

[0437] The preset conditions refer to that the area is within a preset area range, the roundness is equal to a preset roundness value, and the convexity is equal to a preset convexity value.

[0438] Optionally, the processing image acquisition submodule includes:

[0439] a stripe region detection unit, configured to detect a stripe region in the skin region of the candidate size image; wherein the stripe region refers to a connected domain having an area within a second preset area range, a size within a preset length range, and a width within a preset width range;

[0440] a third processed image acquisition unit, configured to, in response to detecting at least one stripe region, remove the stripe region in the candidate size image that meets a second preset condition, to obtain a third processed image;

[0441] a processed image acquisition unit, configured to restore the third processed image to the size of the initial image in response to satisfying the second loop condition, to obtain a final processed image;

[0442] The second cycle condition means that the number of times the stripe regions are detected is equal to a second preset number threshold or the number of the stripe regions is less than a second preset number threshold.

[0443] Optionally, the adjustment image acquisition module includes:

[0444] a mapping relationship acquisition submodule, configured to align the face region in the initial image with a preset skin template to obtain a mapping relationship between the face region and the preset skin template;

[0445] An abnormal region acquisition submodule is configured to, in response to the initial image being sequentially scaled into candidate size images of different preset sizes, acquire abnormal regions of the skin region in the candidate size images to obtain an abnormal region feature map; the abnormal regions include spot regions and / or stripe regions; the spot regions refer to connected domains whose areas are within a first preset area range, whose sizes are smaller than a first preset size threshold and larger than a second preset size threshold; the stripe regions refer to connected domains whose areas are within a second preset area range, whose sizes are within a preset length range, and whose widths are within a preset width range; the second preset area range is larger than the first preset area range;

[0446] The adjusted image acquisition submodule is configured to, in response to the number of resizing operations being equal to a preset number threshold, remove abnormal regions on the last candidate size image to obtain the adjusted image.

[0447] Optionally, the adjusted image acquisition module further includes:

[0448] The second candidate image acquisition submodule is configured to perform scaling processing on the eye region in the first candidate image to obtain a second candidate image, and use the second candidate image as the adjusted image.

[0449] Optionally, the second candidate image acquisition submodule includes:

[0450] a zoom ratio acquisition unit, configured to acquire a zoom ratio of each pixel in the eye area of ​​the first candidate image;

[0451] a target coordinate acquisition unit, configured to acquire target coordinate data of each pixel point according to the zoom ratio and the original coordinate data of each pixel point;

[0452] The second candidate image acquisition unit is configured to replace the original coordinate data of each pixel point in the first candidate image with the target coordinate data to obtain the second candidate image.

[0453] Optionally, the scaling ratio acquiring unit includes:

[0454] a distance ratio acquisition unit, configured to acquire a distance ratio of each pixel point in the eye region of the first candidate image; the distance ratio being used to indicate that the closer the pixel point is to the center point of the eye region, the smaller the moving distance ratio;

[0455] A zoom ratio acquisition unit is used to acquire the zoom ratio of each pixel point according to the distance ratio; the zoom ratio is proportional to the distance ratio.

[0456] Optionally, the distance ratio acquiring unit includes:

[0457] a length acquisition subunit, configured to acquire the major axis length and minor axis length of the eye region;

[0458] The distance ratio acquisition subunit is used to acquire the distance ratio of each pixel point according to the major axis length, the minor axis length, the coordinate data of the center point of the eye area and the original coordinate data of each pixel point.

[0459] Optionally, the scaling ratio acquiring unit includes:

[0460] The zoom ratio acquisition subunit is configured to acquire the zoom ratio of each pixel point according to the distance ratio and the initial value of the zoom ratio.

[0461] Optionally, the second candidate image acquisition submodule further includes:

[0462] The third candidate image acquisition unit is configured to resize the face region in the second candidate image to obtain a third candidate image, and use the third candidate image as the adjusted image.

[0463] Optionally, the third candidate image acquisition unit includes:

[0464] A distance value acquisition subunit, configured to acquire distance values ​​between the control point C1 of the face region and each pixel point in the face region;

[0465] a deformation ratio obtaining subunit, configured to obtain a deformation ratio of each pixel point according to the distance value, wherein the deformation ratio indicates that the farther the pixel point is from the control point C, the smaller the movement distance;

[0466] a target pixel acquisition subunit, configured to calculate target pixel data of each pixel point based on the coordinate data of the target position M of the control point C, the coordinate data of the control point C, and the coordinate data of the target position X of each pixel point;

[0467] The third candidate image acquisition subunit is configured to replace the pixel points of the face area of ​​the second candidate image with the target pixel data to obtain the third candidate image.

[0468] Optionally, the image quality enhancement includes skin resurfacing, whitening, and brightening, and the enhanced image acquisition module includes:

[0469] A first feature map acquisition submodule, configured to perform skin resurfacing processing on the adjusted image to obtain a first skin resurfacing feature map;

[0470] A second feature map acquisition submodule is used to adjust the skin resurfacing degree of the skin resurfacing feature map to obtain a second skin resurfacing feature map;

[0471] a skin color brightening feature map acquisition submodule, configured to perform skin color brightening processing on the second skin resurfacing feature map to obtain a skin color brightening feature map;

[0472] a soft light feature map acquisition submodule, configured to perform soft light processing based on the skin color brightening feature map and the adjusted image to obtain a soft light feature map;

[0473] The enhanced image acquisition submodule is used to acquire the enhanced image according to the adjusted image, the second skin resurfacing feature map, the skin color brightening feature map, the soft light feature map and their respective weight values.

[0474] Optionally, the first feature map acquisition submodule includes:

[0475] a filtered sub-image obtaining unit, configured to perform filtering processing on the green sub-image of the adjusted image to obtain a filtered sub-image;

[0476] a difference sub-image acquisition unit, configured to acquire a difference in pixel data of a same pixel point in the adjusted image and the filtered sub-image to obtain a difference sub-image;

[0477] a compensation sub-image acquisition unit, configured to compensate pixel data of each pixel point in the difference sub-image based on preset compensation data to obtain the difference compensation sub-image;

[0478] The first feature map acquisition unit is configured to perform light compensation on the difference compensation sub-image to obtain the first skin resurfacing feature map.

[0479] Optionally, the filtering method used for filtering the green sub-image of the adjusted image includes one of the following: averaging filtering, Gaussian filtering, bilateral filtering and guided filtering.

[0480] Optionally, the filter kernel of the Gaussian filter is implemented using a circular filter kernel, and the sum of the weight values ​​of the circular filter kernel is equal to a preset weight value.

[0481] Optionally, the first feature map acquiring unit includes:

[0482] a pixel data comparison subunit, configured to compare pixel data of each pixel point of the difference-compensated sub-image with preset compensation data;

[0483] a target pixel data acquisition subunit, configured to acquire target pixel data corresponding to the pixel data according to a first light compensation model when the pixel data of each pixel point is greater than or equal to the preset compensation data;

[0484] The target pixel data acquisition subunit is further configured to acquire target pixel data corresponding to the pixel data according to a second light compensation model when the pixel data of each pixel point is less than the preset compensation data;

[0485] a pixel data replacement subunit, configured to replace pixel data of each pixel point of the difference-compensated sub-image with the target pixel data;

[0486] The first feature map acquisition subunit is configured to determine, in response to the number of compensation times reaching a preset compensation times threshold, that the difference compensation sub-image obtained by the last compensation is the first skin resurfacing feature map.

[0487] Optionally, the preset compensation times threshold value has a value range of [1, 10].

[0488] Optionally, the second feature map acquisition submodule includes:

[0489] A grayscale image acquisition unit, configured to acquire a grayscale image corresponding to the adjusted image;

[0490] a brightened image acquisition unit, configured to perform a brightening process on the grayscale data of each pixel in the grayscale image based on preset skin resurfacing degree data to obtain a grayscale brightened image;

[0491] The second feature map acquiring unit is configured to adjust the pixel data of each pixel point of the adjusted image according to the first skin resurfacing feature map and the grayscale brightened image to obtain the second skin resurfacing feature map.

[0492] Optionally, after performing recognition processing on each frame of the video image of the video stream, the apparatus further includes:

[0493] The enhanced image acquisition module is further configured to, in response to no human face region being detected in the video image, perform image quality enhancement processing on the video image to obtain an enhanced image;

[0494] The target image acquisition module is further used to render the enhanced image to obtain a target image.

[0495] It should be noted that the device embodiment shown in this embodiment matches the content of the above method embodiment. You can refer to the content of the above method embodiment and will not repeat it here.

[0496] In an exemplary embodiment, an electronic device is also provided, comprising

[0497] Display screen;

[0498] processor;

[0499] a memory for storing a computer program executable by the processor;

[0500] The processor is configured to execute the computer program in the memory to implement the above method.

[0501] In an exemplary embodiment, a server is also provided, comprising

[0502] processor;

[0503] a memory for storing a computer program executable by the processor;

[0504] The processor is configured to execute the computer program in the memory to implement the above method.

[0505] In an exemplary embodiment, a video conferencing system is further provided, comprising a plurality of electronic devices and a plurality of servers; the servers comprising:

[0506] processor;

[0507] a memory for storing a computer program executable by the processor;

[0508] The processor is configured to execute the computer program in the memory to implement the above method.

[0509] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including an executable computer program. The executable computer program can be executed by a processor to implement the method of the above embodiment. The computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0510] Other embodiments of the present disclosure will readily occur to those skilled in the art after considering the specification and practicing the disclosure herein. This disclosure is intended to cover any variations, uses, or adaptations that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the following claims.

[0511] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. An image processing method, characterized in that: The method comprises: Perform recognition processing on each frame of the video stream to obtain an initial image containing a human face; Performing an alignment process on the face region in the initial image and the preset skin template to obtain a mapping relationship between the face region and the preset skin template; Adjusting the skin area of ​​the face area to obtain an adjusted image; the skin area is determined according to the mapping relationship; Performing image quality enhancement processing on the adjusted image to obtain an enhanced image; The enhanced image is rendered to obtain a target image.

2. The method according to claim 1, characterized in that Performing an alignment process on the face region in the initial image and the preset skin template to obtain a mapping relationship between the face region and the preset skin template includes: Obtaining the coordinates of key points of the face in the preset skin template to obtain a first key point coordinate vector; Obtaining the coordinates of key points of the face in the initial image to obtain a second key point coordinate vector; The first key point coordinate vector and the second key point coordinate vector are input into a preset distance model and a solution of the preset distance model is obtained, and the solution of the preset distance model is used as the mapping relationship.

3. The method according to claim 2, characterized in that Obtaining a solution to the preset distance model includes: Respectively obtain the coordinate mean and coordinate standard deviation of the first key point coordinate vector to obtain a first coordinate mean and a first coordinate standard deviation; Respectively obtaining a coordinate mean and a coordinate standard deviation of the coordinate vector of the second key point to obtain a second coordinate mean and a second coordinate standard deviation; Subtract the first coordinate mean from the first key point coordinate vector and divide the resultant value by the first coordinate standard deviation to obtain a first adjusted coordinate vector; Subtract the second coordinate mean from the second key point coordinate vector and divide the resultant value by the second coordinate standard deviation to obtain a second adjusted coordinate vector; The first adjusted coordinate vector and the second adjusted coordinate vector are input into the preset distance model and singular value decomposition is performed on the preset distance model to obtain the mapping relationship.

4. The method according to claim 1, characterized in that: Adjusting the skin area of ​​the face area to obtain an adjusted image includes: In response to successively scaling the initial image by a preset size ratio to obtain a candidate size image, clearing the candidate size image to obtain a processed image; In response to the number of resizing being equal to a preset number threshold, the last processed image is restored to the size of the initial image to obtain a first candidate image, and the first candidate image is used as the adjusted image.

5. The method according to claim 4, characterized in that Performing a clearing process on the candidate size image to obtain a processed image includes: Detecting at least one spot region of the skin region of the candidate size image; the spot region refers to a connected domain whose area is within a first preset area range and whose size is smaller than a first preset size threshold and larger than a second preset size threshold; performing a clearing process on the at least one spot region to obtain a first processed image; In response to satisfying the first loop condition, restoring the first processed image to the size of the initial image to obtain a final processed image; The first cycle condition means that the number of times the spot regions are detected is equal to a first preset number threshold or the number of the spot regions is less than a first preset number threshold.

6. The method according to claim 5, characterized in that Detecting at least one spot region of the skin region of the candidate size image, comprising: Acquire target pixel data according to the mapping relationship and pixel data of each pixel point of the preset skin template; The target pixel data is used to replace the pixel data at the corresponding position in the candidate size image to obtain the second candidate image; Connected domain detection is performed on the skin area of ​​the second candidate image to obtain a connected domain whose spot area size of the skin area is smaller than a first preset size threshold and larger than a second preset size threshold as the spot area of ​​the skin area.

7. The method according to claim 6, characterized in that Performing connected domain detection on the skin area of ​​the second candidate image to obtain a spot area of ​​the skin area includes: Convert the grayscale image into a binary image according to each preset threshold value to obtain multiple frames of binary images; Respectively obtain the connected areas of the binary images of each frame to obtain multiple connected domains of the binary images of each frame; Classifying the multiple connected domains according to the distances between the center points of each connected domain of the multiple frames of binary images, and obtaining multiple connected domains whose distances between the center points are less than a preset distance threshold; The positions and sizes of the multiple connected domains are acquired, and the connected domains whose sizes are smaller than a first preset size threshold and larger than a second preset size threshold are taken as the spot regions of the skin region.

8. The method according to claim 5, characterized in that After detecting at least one spot region of the skin region of the candidate size image, the method further comprises: In response to not detecting the spot area, determining whether there is a stripe area; wherein the stripe area refers to a connected domain whose area is within a second preset area range, whose size is within a preset length range, and whose width is within a preset width range; In response to detecting at least one stripe region, clearing the stripe region in the candidate size image that meets a second preset condition to obtain a second processed image; In response to satisfying the second loop condition, restoring the second processed image to the size of the initial image to obtain a final processed image; The second cycle condition means that the number of times the stripe regions are detected is equal to a second preset number threshold or the number of the stripe regions is less than a second preset number threshold.

9. The method according to claim 4, characterized in that Performing a clearing process on the candidate size image to obtain a processed image includes: Detecting a stripe region of the skin region of the candidate size image; wherein the stripe region refers to a connected domain whose area is within a second preset area range, whose size is within a preset length range, and whose width is within a preset width range; In response to detecting at least one stripe region, clearing the stripe region in the candidate size image that meets a second preset condition to obtain a third processed image; In response to satisfying the second loop condition, restoring the third processed image to the size of the initial image to obtain a final processed image; The second cycle condition means that the number of times the stripe regions are detected is equal to a second preset number threshold or the number of the stripe regions is less than a second preset number threshold.

10. The method according to claim 1, characterized in that Performing face adjustment processing on the face region in the initial image to obtain an adjusted image includes: Performing an alignment process on the face region in the initial image and the preset skin template to obtain a mapping relationship between the face region and the preset skin template; In response to the initial image being scaled into candidate size images of different preset sizes in sequence, an abnormal area of ​​the skin area in the candidate size image is obtained to obtain an abnormal area feature map; the abnormal area includes a spot area and / or a stripe area; the spot area refers to a connected domain whose area is within a first preset area range, whose size is less than a first preset size threshold and greater than a second preset size threshold; the stripe area refers to a connected domain whose area is within a second preset area range, whose size is within a preset length range and whose width is within a preset width range; the second preset area range is larger than the first preset area range; In response to the number of resizing being equal to a preset number threshold, the abnormal processing area is cleared on the last candidate size image to obtain the adjusted image.

11. The method according to claim 4, characterized in that After acquiring the first candidate image, the method further includes: The eye region in the first candidate image is scaled to obtain a second candidate image, and the second candidate image is used as the adjusted image.

12. The method according to claim 11, characterized in that Scaling the eye region in the first candidate image to obtain a second candidate image includes: Obtaining a scaling ratio of each pixel in the eye area of ​​the first candidate image; Acquire target coordinate data of each pixel point according to the zoom ratio and the original coordinate data of each pixel point; The original coordinate data of each pixel in the first candidate image is replaced with the target coordinate data to obtain the second candidate image.

13. The method according to claim 12, characterized in that Obtaining a scaling ratio of each pixel point in the eye area of ​​the first candidate image includes: Obtaining a distance ratio of each pixel point in the eye region of the first candidate image; the distance ratio is used to indicate that the closer the pixel point is to the center point of the eye region, the smaller the moving distance ratio; The scaling ratio of each pixel point is obtained according to the distance ratio; the scaling ratio is proportional to the distance ratio.

14. The method according to claim 13, characterized in that Obtaining the distance ratio of each pixel point in the eye area of ​​the first candidate image includes: Obtaining the major axis length and minor axis length of the eye region; The distance ratio of each pixel point is obtained according to the long axis length, the short axis length, the coordinate data of the center point of the eye area and the original coordinate data of each pixel point.

15. The method according to claim 13, characterized in that Obtaining the scaling ratio of each pixel point according to the distance ratio includes: The zoom ratio of each pixel point is obtained according to the distance ratio and the initial value of the zoom ratio.

16. The method according to any one of claims 11 to 15, characterized in that: After acquiring the second candidate image, the method further includes: The face region in the second candidate image is resized to obtain a third candidate image, and the third candidate image is used as the adjusted image.

17. The method according to claim 16, characterized in that Resizing the face region in the second candidate image to obtain a third candidate image includes: Obtaining distance values ​​between a control point C1 of a face region and each pixel point in the face region; Acquire a deformation ratio of each pixel point according to the distance value, wherein the deformation ratio is used to indicate that the farther the pixel point is from the control point C, the smaller the movement distance is; Calculating target pixel data of each pixel point according to the coordinate data of the target position M of the control point C, the coordinate data of the control point C and the coordinate data of the target position X of each pixel point; The pixel points of the face area of ​​the second candidate image are replaced with the target pixel data to obtain the third candidate image.

18. The method according to claim 1, characterized in that The image quality enhancement includes skin smoothing, whitening and brightening. The image quality enhancement is performed on the adjusted image to obtain an enhanced image, including: Performing skin resurfacing processing on the adjusted image to obtain a first skin resurfacing feature map; adjusting the skin resurfacing degree of the skin resurfacing feature map to obtain a second skin resurfacing feature map; performing skin lightening processing on the second skin resurfacing feature map to obtain a skin lightening feature map; Performing soft light processing based on the skin color brightening feature map and the adjusted image to obtain a soft light feature map; The enhanced image is obtained according to the adjusted image, the second skin resurfacing feature map, the skin color brightening feature map, the soft light feature map, and their respective weight values.

19. The method according to claim 18, characterized in that Performing skin grinding processing on the adjusted image to obtain a first skin grinding feature map includes: Performing filtering processing on the green sub-image of the adjusted image to obtain a filtered sub-image; Obtaining a pixel data difference of the same pixel point in the adjusted image and the filtered sub-image to obtain a difference sub-image; Compensating pixel data of each pixel point in the difference sub-image based on preset compensation data to obtain the difference compensated sub-image; Light compensation is performed on the difference compensation sub-image to obtain the first skin resurfacing feature map.

20. The method according to claim 19, characterized in that The filtering method used for filtering the green sub-image of the adjusted image includes one of the following: averaging filtering, Gaussian filtering, bilateral filtering and guided filtering.

21. The method according to claim 20, characterized in that The filter kernel of the Gaussian filter is implemented by a circular filter kernel, and the sum of the weight values ​​of the circular filter kernel is equal to the preset weight value.

22. The method according to claim 19, characterized in that Performing light compensation on the difference compensation sub-image to obtain the first skin resurfacing feature map includes: Comparing pixel data of each pixel point of the difference compensation sub-image with preset compensation data; When the pixel data of each pixel point is greater than or equal to the preset compensation data, acquiring target pixel data corresponding to the pixel data according to the first light compensation model; When the pixel data of each pixel point is less than the preset compensation data, acquiring target pixel data corresponding to the pixel data according to a second light compensation model; Replacing pixel data of each pixel point of the difference compensation sub-image with the target pixel data; In response to the number of compensation times reaching a preset compensation times threshold, the difference compensation sub-image obtained by the last compensation is determined as the first skin resurfacing feature image.

23. The method according to claim 21, characterized in that The preset compensation times threshold value has a value range of [1, 10].

24. The method according to claim 18, characterized in that Adjusting the skin resurfacing degree of the skin resurfacing feature map to obtain a second skin resurfacing feature map includes: Obtaining a grayscale image corresponding to the adjusted image; Brightening the grayscale data of each pixel in the grayscale image based on the preset skin resurfacing degree data to obtain a grayscale brightened image; The pixel data of each pixel point of the adjusted image is adjusted according to the first skin resurfacing feature map and the grayscale brightened image to obtain the second skin resurfacing feature map.

25. The method according to claim 1, characterized in that After performing recognition processing on each frame of the video image of the video stream, the method further includes: In response to no human face region being detected in the video image, performing image quality enhancement processing on the video image to obtain an enhanced image; The enhanced image is rendered to obtain a target image.

26. An image processing device, characterized in that: The device comprises: The initial image acquisition module is used to perform recognition processing on each frame of the video stream to obtain an initial image containing a human face; A mapping relationship acquisition module, used to align the face area in the initial image with the preset skin template to obtain a mapping relationship between the face area and the preset skin template; An adjusted image acquisition module, used to adjust the skin area of ​​the face area to obtain an adjusted image; the skin area is determined according to the mapping relationship; An enhanced image acquisition module, used for performing image quality enhancement processing on the adjusted image to obtain an enhanced image; The target image acquisition module is used to render the enhanced image to obtain a target image.

27. The device according to claim 26, characterized in that The mapping relationship acquisition module includes: The first vector acquisition submodule is used to obtain the key point coordinates of the face in the preset skin template, and obtain the first Keypoint coordinate vector; A second vector acquisition submodule is used to acquire the key point coordinates of the face in the initial image to obtain a second key point coordinate vector; A mapping relationship acquisition submodule is used to input the first key point coordinate vector and the second key point coordinate vector into a preset distance model and obtain a solution of the preset distance model, and use the solution of the preset distance model as the mapping relationship.

28. The device according to claim 27, characterized in that The mapping relationship acquisition submodule includes: A first vector processing unit, used to respectively obtain a coordinate mean and a coordinate standard deviation of the first key point coordinate vector to obtain a first coordinate mean and a first coordinate standard deviation; A second vector processing unit, used to respectively obtain a coordinate mean and a coordinate standard deviation of the second key point coordinate vector to obtain a second coordinate mean and a second coordinate standard deviation; A first vector adjustment unit, configured to subtract the first coordinate mean from the first key point coordinate vector and divide the result by the first coordinate standard deviation to obtain a first adjusted coordinate vector; A second vector adjustment unit, configured to subtract the second coordinate mean from the second key point coordinate vector and divide the resultant value by the second coordinate standard deviation to obtain a second adjusted coordinate vector; A mapping relationship acquisition unit is used to input the first adjusted coordinate vector and the second adjusted coordinate vector into the preset distance model and perform singular value decomposition on the preset distance model to obtain the mapping relationship.

29. The device according to claim 26, characterized in that The image adjustment acquisition module comprises: A processed image acquisition submodule, configured to, in response to scaling the initial image in sequence to a candidate size image of a preset size ratio, perform a clearing process on the candidate size image to obtain a processed image; The first candidate image acquisition submodule is used to restore the last processed image to the size of the initial image in response to the number of resizing being equal to a preset number threshold, to obtain a first candidate image, and to use the first candidate image as the adjusted image.

30. The device according to claim 29, characterized in that The processing image acquisition submodule comprises: A spot region detection unit, configured to detect at least one spot region of the skin region of the candidate size image; the spot region refers to a connected domain whose area is within a first preset area range and whose size is smaller than a first preset size threshold and larger than a second preset size threshold; A first processed image acquisition unit, configured to perform a clearing process on the at least one spot region to obtain a first processed image; a processed image acquisition unit, configured to restore the first processed image to the size of the initial image in response to satisfying the first loop condition, so as to obtain a final processed image; The first cycle condition means that the number of times the spot regions are detected is equal to a first preset number threshold or the number of the spot regions is less than a first preset number threshold.

31. The device according to claim 30, characterized in that The spot area detection unit comprises: A target pixel acquisition subunit, configured to acquire target pixel data according to the mapping relationship and pixel data of each pixel point of the preset skin template; A second candidate image acquisition subunit is used to replace the pixel data at the corresponding position in the candidate size image with the target pixel data to obtain a second candidate image; The spot area acquisition subunit is used to perform connected domain detection on the skin area of ​​the second candidate image, and obtain a connected domain whose spot area size of the skin area is smaller than a first preset size threshold and larger than a second preset size threshold as the spot area of ​​the skin area.

32. The device according to claim 31, characterized in that The spot area acquisition subunit comprises: A binary image acquisition subunit is used to convert the grayscale image into a binary image according to each preset threshold value to obtain multiple frames of binary images; A first connected domain acquisition subunit is used to respectively acquire connected areas of the binary images of each frame to obtain multiple connected domains of the binary images of each frame; A second connected domain acquisition subunit is used to classify the multiple connected domains according to the distances between the center points of each connected domain of the multiple frames of binary images, and obtain multiple connected domains whose distances between the center points are less than a preset distance threshold; The spot area acquisition subunit is used to obtain the positions and sizes of the multiple connected domains, and use the connected domains whose sizes are smaller than a first preset size threshold and larger than a second preset size threshold as the spot area of ​​the skin area; the position refers to the average value of the center point coordinates of the connected domains of each frame of the binary image; the size refers to the radius length of the circumscribed circle when the area of ​​the connected domain of the binary image is within a preset area range.

33. The device according to claim 30, characterized in that After detecting at least one spot area of ​​the skin area of ​​the candidate size image, the processing image acquisition submodule further includes: a stripe region determination unit, for determining whether a stripe region exists in response to the detection of no spot region; wherein the stripe region refers to a connected domain whose area is within a second preset area range, whose size is within a preset length range, and whose width is within a preset width range; a spot region clearing unit, configured to, in response to detecting at least one stripe region, clear the stripe region in the candidate size image that meets a second preset condition to obtain a second processed image; a processed image acquisition unit, configured to restore the second processed image to the size of the initial image in response to satisfying the second loop condition, so as to obtain a final processed image; The second cycle condition means that the number of times the stripe regions are detected is equal to a second preset number threshold or the number of the stripe regions is less than a second preset number threshold.

34. The device according to claim 29, characterized in that The processing image acquisition submodule comprises: a stripe region detection unit, configured to detect a stripe region of a skin region of the candidate size image; wherein the stripe region refers to a connected domain whose area is within a second preset area range, whose size is within a preset length range, and whose width is within a preset width range; A third processed image acquisition unit is used for clearing the stripe region that meets the second preset condition in the candidate size image in response to detecting at least one stripe region, so as to obtain a third processed image; a processed image acquisition unit, configured to restore the third processed image to the size of the initial image in response to satisfying the second loop condition, so as to obtain a final processed image; The second cycle condition means that the number of times the stripe regions are detected is equal to a second preset number threshold or the number of the stripe regions is less than a second preset number threshold.

35. The device according to claim 26, characterized in that The image adjustment acquisition module comprises: A mapping relationship acquisition submodule, used for aligning the face area in the initial image with the preset skin template to obtain a mapping relationship between the face area and the preset skin template; The abnormal region acquisition submodule is used for, in response to the initial image being scaled sequentially into candidate size images of different preset sizes, acquiring the abnormal region of the skin region in the candidate size image, and obtaining an abnormal region feature map; the abnormal region includes a spot region and / or a stripe region; the spot region refers to a connected domain whose area is within a first preset area range, whose size is less than a first preset size threshold and greater than a second preset size threshold; the stripe region refers to a connected domain whose area is within a second preset area range, whose size is within a preset length range and whose width is within a preset width range; the second preset area range is larger than the first preset area range; The adjusted image acquisition submodule is used to clear the abnormal area on the last candidate size image in response to the number of resizing being equal to a preset number threshold, so as to obtain the adjusted image.

36. The device according to claim 29, characterized in that The adjustment image acquisition module also includes: The second candidate image acquisition submodule is used to perform scaling processing on the eye region in the first candidate image to obtain a second candidate image, and use the second candidate image as the adjusted image.

37. The device according to claim 36, characterized in that The second candidate image acquisition submodule comprises: a zoom ratio acquisition unit, configured to acquire a zoom ratio of each pixel in the eye region of the first candidate image; A target coordinate acquisition unit is used to acquire the original coordinate data of each pixel point according to the zoom ratio. Target coordinate data of each pixel point; The second candidate image acquisition unit is used to replace the original coordinate data of each pixel in the first candidate image with the target coordinate data to obtain the second candidate image.

38. The device according to claim 37, characterized in that The zoom ratio acquisition unit comprises: a distance ratio acquisition unit, configured to acquire a distance ratio of each pixel point in the eye region of the first candidate image; the distance ratio is used to indicate that the closer the pixel point is to the center point of the eye region, the smaller the moving distance ratio; A zoom ratio acquisition unit is used to acquire the zoom ratio of each pixel point according to the distance ratio; the zoom ratio is proportional to the distance ratio.

39. The device according to claim 38, characterized in that The distance ratio acquisition unit comprises: A length acquisition subunit, used to acquire the major axis length and the minor axis length of the eye region; The distance ratio acquisition subunit is used to acquire the distance ratio of each pixel point according to the major axis length, the minor axis length, the coordinate data of the center point of the eye area and the original coordinate data of each pixel point.

40. The device according to claim 38, characterized in that The zoom ratio acquisition unit comprises: The zoom ratio acquisition subunit is used to acquire the zoom ratio of each pixel point according to the distance ratio and the initial value of the zoom ratio.

41. The device according to any one of claims 36 to 40, characterized in that The second candidate image acquisition submodule further includes: The third candidate image acquisition unit is used to resize the face area in the second candidate image to obtain a third candidate image, and use the third candidate image as the adjusted image.

42. The device according to claim 41, characterized in that The third candidate image acquisition unit comprises: A distance value acquisition subunit, used to acquire the distance value between the control point C1 of the face area and each pixel point in the face area; A deformation ratio acquisition subunit, used for acquiring a deformation ratio of each pixel point according to the distance value, wherein the deformation ratio is used to indicate that the farther the pixel point is from the control point C, the smaller the movement distance is; A target pixel acquisition subunit, used for calculating the target pixel data of each pixel point according to the coordinate data of the target position M of the control point C, the coordinate data of the control point C and the coordinate data of the target position X of each pixel point; The third candidate image acquisition subunit is used to replace the pixel points of the face area of ​​the second candidate image with the target pixel data to obtain the third candidate image.

43. The device according to claim 26, characterized in that The image quality enhancement includes skin resurfacing, whitening and brightening, and the enhanced image acquisition module includes: A first feature map acquisition submodule, configured to perform skin resurfacing processing on the adjusted image to obtain a first skin resurfacing feature map; A second feature map acquisition submodule, used for adjusting the skin resurfacing degree of the skin resurfacing feature map to obtain a second skin resurfacing feature map; A skin color brightening feature map acquisition submodule, used for performing skin color brightening processing on the second skin resurfacing feature map to obtain a skin color brightening feature map; A soft light feature map acquisition submodule, used for performing soft light processing based on the skin color brightening feature map and the adjusted image to obtain a soft light feature map; The enhanced image acquisition submodule is used to acquire the enhanced image according to the adjusted image, the second skin resurfacing feature map, the skin color brightening feature map and the soft light feature map and their respective weight values.

44. The device according to claim 43, characterized in that The first feature map acquisition submodule includes: A filtered sub-image acquisition unit, used for performing filtering processing on the green sub-image of the adjusted image to obtain a filtered sub-image; A difference sub-image acquisition unit, used to acquire a pixel data difference of the same pixel point in the adjusted image and the filtered sub-image to obtain a difference sub-image; A compensation sub-image acquisition unit, used for compensating the pixel data of each pixel point in the difference sub-image based on preset compensation data to obtain the difference compensation sub-image; The first feature map acquisition unit is used to perform light compensation on the difference compensation sub-image to obtain the first skin resurfacing feature map.

45. The device according to claim 44, characterized in that The filtering method used for filtering the green sub-image of the adjusted image includes one of the following: averaging filtering, Gaussian filtering, bilateral filtering and guided filtering.

46. ​​The device according to claim 45, characterized in that The filter kernel of the Gaussian filter is implemented by a circular filter kernel, and the sum of the weight values ​​of the circular filter kernel is equal to the preset weight value.

47. The device according to claim 46, characterized in that The first feature map acquisition unit includes: A pixel data comparison subunit, used for comparing the pixel data of each pixel point of the difference compensation sub-image with preset compensation data; a target pixel data acquisition subunit, configured to acquire target pixel data corresponding to the pixel data according to a first light compensation model when the pixel data of each pixel point is greater than or equal to the preset compensation data; The target pixel data acquisition subunit is further configured to acquire target pixel data corresponding to the pixel data according to a second light compensation model when the pixel data of each pixel point is less than the preset compensation data; A pixel data replacement subunit, used to replace the pixel data of each pixel point of the difference compensation sub-image with the target pixel data; The first feature map acquisition subunit is used to determine, in response to the number of compensation times reaching a preset compensation times threshold, that the difference compensation sub-image obtained by the last compensation is the first skin resurfacing feature map.

48. The device according to claim 46, characterized in that The preset compensation times threshold value has a value range of [1, 10].

49. The device according to claim 43, characterized in that The second feature map acquisition submodule includes: A grayscale image acquisition unit, used to acquire a grayscale image corresponding to the adjusted image; a brightened image acquisition unit, configured to perform brightening processing on the grayscale data of each pixel in the grayscale image based on preset skin resurfacing degree data to obtain a grayscale brightened image; The second feature map acquisition unit is used to adjust the pixel data of each pixel point of the adjusted image according to the first skin resurfacing feature map and the grayscale brightened image to obtain the second skin resurfacing feature map.

50. The device according to claim 26, characterized in that After performing recognition processing on each frame of the video image of the video stream, the device further includes: The enhanced image acquisition module is further used to perform image quality enhancement processing on the video image to obtain an enhanced image in response to no face region being detected in the video image; The target image acquisition module is also used to render the enhanced image to obtain the target image.

51. An electronic device, characterized in that: include Display screen; processor; a memory for storing a computer program executable by the processor; The processor is configured to execute the computer program in the memory to implement the method according to any one of claims 1 to 25.

52. A computer-readable storage medium, characterized in that: When the executable computer program in the storage medium is executed by a processor, the method according to any one of claims 1 to 25 can be implemented.

Images