Image processing method, device, equipment and storage medium

Abstract

The embodiment of the present invention discloses an image processing method, device, equipment and storage medium. The method includes: obtaining an instruction to adjust a set face area in an original face image according to a reference face image; according to the instruction, obtaining an original region face key point set and a reference region face key point set matching the face area in the original face image and the reference face image respectively; according to the reference region face key point set, determining an adjusted region face key point set matching the original region face key point set; according to the adjusted region face key point set, adjusting the original grid in the original face image to generate an adjusted face image. The technical solution of the embodiment of the present invention does not limit the size and shape of the original face image and the adjusted face image, improves the deformation adjustment speed of the original face image, and can be adapted to real-time face transformation of videos.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of image processing technology, and in particular, to an image processing method, apparatus, device and storage medium. Background Art

[0002] With the development of society, electronic devices such as mobile phones and tablet computers have been widely used in learning, entertainment, work, etc. Many electronic devices are equipped with cameras for taking pictures, recording videos, live broadcasting, etc. If the image data of the camera contains a face, users often adjust the face, such as beautification, face changing, etc.

[0003] At present, when adjusting the face shape or facial features of a human face, only when the size or shape of the original face image and the adjusted face image are similar, can the adjusted face image be compared with the original face image to perform a corresponding deformation to obtain the adjusted face image. In the process of realizing the present invention, the inventor found that the prior art has the following defects: there are restrictions on the size and shape of the original face image and the adjusted face image, it is impossible to adjust the face in a targeted manner, the adjustment efficiency is low, and it cannot adapt to real-time face transformation of the video. Summary of the invention

[0004] The embodiments of the present invention provide an image processing method, apparatus, device and storage medium to achieve no restriction on the size and shape of the original face image and the adjusted face image, improve the deformation adjustment speed of the original face image, and adapt to real-time face transformation of video.

[0005] In a first aspect, an embodiment of the present invention provides an image processing method, including:

[0006] Obtaining an instruction to adjust a set face area in an original face image according to a reference face image; The original face image is composed of a plurality of original grids divided by face key points;

[0007] According to the instruction, respectively obtaining a set of original area face key points and a set of reference area face key points matching the face area in the original face image and the reference face image;

[0008] Determine, based on the reference regional face key point set, an adjusted regional face key point set that matches the original regional face key point set;

[0009] According to the set of facial key points in the adjusted area, the original grid in the original face image is adjusted to generate an adjusted face image.

[0010] Optionally, determining, based on the reference regional face key point set, an adjusted regional face key point set that matches the original regional face key point set includes:

[0011] Determine target fixed points in the original face image and the reference face image respectively;

[0012] The position vectors between the face key points of each region and the corresponding target fixed points in the face key point set of the reference region and the face key point set of the original region are calculated respectively, and a plurality of position vector pairs are formed;

[0013] Calculate the adjusted position vector corresponding to each regional face key point in the original regional face key point set according to the preset change strength and each position vector pair;

[0014] According to the adjusted position vector, a set of adjusted regional face key points matching the set of original regional face key points is determined.

[0015] Optionally, determining target fixed points in the original face image and the reference face image respectively includes:

[0016] Determine the minimum bounding rectangles corresponding to the original face image and the reference face image, respectively, based on the sets of face key points corresponding to the original face image and the reference face image, respectively;

[0017] In each minimum bounding rectangle, obtain the corner point at the same position as the target fixed point.

[0018] Optionally, determining the minimum bounding rectangles corresponding to the original face image and the reference face image respectively according to the sets of face key points corresponding to the original face image and the reference face image respectively includes:

[0019] In the facial key point sets corresponding to the original facial image and the reference facial image, respectively, the maximum value points in four directions of the image coordinate system are obtained;

[0020] According to the maximum value points in four directions of the image coordinate system corresponding to each set of facial key points, the minimum circumscribed rectangles corresponding to the original facial image and the reference facial image are determined.

[0021] Optionally, according to the preset change strength and each position vector pair, calculating the adjusted position vector corresponding to each regional face key point in the original regional face key point set includes:

[0022] Calculating a scaling factor between the original face image and the reference face image based on the minimum bounding rectangles corresponding to the original face image and the reference face image respectively;

[0023] According to the preset change strength, each position vector pair, and the scaling factor, the adjusted position vector corresponding to each regional face key point in the original regional face key point set is calculated.

[0024] Optionally, according to a preset change strength, each position vector pair, and a scaling factor, an adjusted position vector corresponding to each regional face key point in the original regional face key point set is calculated, including:

[0025] According to the formula: σ*XnA+(1-σ)*Xn'A'*K=XnzAz, the adjusted position vector XnzAz of the regional face key point Xn' in the original regional face key point set is calculated, and XnzAz corresponds to the nth position vector pair (XnA, Xn'A');

[0026] Among them, XnA is the position vector between the regional face key point Xn in the reference regional face key point set and the target fixed point A in the reference face image, Xn'A' is the position vector between the regional face key point Xn' in the original regional face key point set and the target fixed point A' in the original face image, Xn matches Xn', σ is the change intensity, σ∈[0,1], and K is the scaling factor.

[0027] Optionally, adjusting the original grid in the original face image according to the set of face key points in the adjusted region to generate the adjusted face image includes:

[0028] Create a blank image that matches the original face image;

[0029] Determine the grid deformation mode of each original grid in the original face image according to the set of face key points in the adjusted region;

[0030] According to the grid deformation mode, a plurality of target deformed grids corresponding to the original grid are divided in the blank image;

[0031] According to the position correspondence between the original grid and the target deformed grid, each pixel in the original grid is mapped to the corresponding target deformed grid to obtain an adjusted face image.

[0032] Optionally, according to the position correspondence between the original grid and the target deformed grid, each pixel point in the original grid is mapped to the corresponding target deformed grid to obtain an adjusted face image, including:

[0033] Get an original grid in the original face image as the current processing grid;

[0034] In the blank image, obtain the target deformed mesh that matches the current processing mesh as the paired mesh;

[0035] Obtaining a first vertex sequence corresponding to the currently processed mesh and a second vertex sequence corresponding to the paired mesh, and calculating a mapping relationship matrix between the currently processed mesh and the paired mesh according to the first vertex sequence and the second vertex sequence;

[0036] According to the mapping relationship matrix, each pixel point in the current processing grid is mapped to the paired grid, and the operation of obtaining an original grid in the original face image as the current processing grid is returned to be executed until the processing of all the original grids is completed.

[0037] Optionally, before obtaining the instruction to adjust the set face area in the original face image according to the reference face image, the method further includes:

[0038] Acquire facial image data, and identify facial key points in the facial image data;

[0039] According to the face association points, the face image data is divided into a plurality of grids, wherein the vertices of the grids at least include the face key points;

[0040] Draw the grids, obtain the original face image for display, and use the drawn grids as the original grids.

[0041] In a second aspect, an embodiment of the present invention further provides an image processing device, including:

[0042] An adjustment instruction acquisition module is used to acquire an instruction to adjust a set face area in an original face image according to a reference face image; the original face image is composed of a plurality of original grids divided by face key points;

[0043] A key point set acquisition module is used to acquire, according to the instruction, an original area face key point set and a reference area face key point set matching the face area in the original face image and the reference face image respectively;

[0044] A determination module, used to determine, based on the reference regional face key point set, a set of adjusted regional face key points that matches the original regional face key point set;

[0045] The adjustment module is used to adjust the original grid in the original face image according to the set of face key points in the adjusted area to generate an adjusted face image.

[0046] In a third aspect, an embodiment of the present invention further provides a device, the device comprising:

[0047] one or more processors;

[0048] a storage device for storing one or more programs,

[0049] When one or more programs are executed by one or more processors, the one or more processors implement the image processing method provided by any embodiment of the present invention.

[0050] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the image processing method provided by any embodiment of the present invention.

[0051] In an embodiment of the present invention, an instruction for adjusting a set face area in an original face image according to a reference face image is obtained; the original face image is composed of a plurality of original grids divided by face key points; according to the instruction, an original region face key point set and a reference region face key point set matching the face area are obtained in the original face image and the reference face image respectively; according to the reference region face key point set, an adjusted region face key point set matching the original region face key point set is determined; according to the adjusted region face key point set, the original grid in the original face image is adjusted to generate an adjusted face image, thereby solving the problems in the prior art of limiting the size and shape of the original face image and the adjusted face image and low face adjustment efficiency, not limiting the size and shape of the original face image and the adjusted face image, improving the deformation adjustment speed of the original face image, and being adaptable to real-time face transformation of videos. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1a is a flow chart of an image processing method in Embodiment 1 of the present invention;

[0053] Figure 1b is a schematic diagram of key points of a face in Embodiment 1 of the present invention;

[0054] Figure 1c is a schematic diagram of an original grid in a face image in Embodiment 1 of the present invention;

[0055] Figure 1d This is a schematic diagram of grid pairing in Embodiment 1 of the present invention;

[0056] Figure 2a is a flow chart of an image processing method in Embodiment 2 of the present invention;

[0057] Figure 2b is a schematic diagram of the minimum circumscribed rectangle in the second embodiment of the present invention;

[0058] Figure 3 is a structural schematic diagram of an image processing device in Embodiment 3 of the present invention;

[0059] Figure 4 It is a structural diagram of a device in Embodiment 4 of the present invention. DETAILED DESCRIPTION

[0060] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0061] Embodiment 1

[0062] Figure 1a is a flow chart of an image processing method in the first embodiment of the present invention. This embodiment is applicable to the case of fast real-time deformation adjustment of a face in an image. The method can be executed by an image processing device, which can be implemented by hardware and / or software and can generally be integrated in a device that provides image processing services. Figure 1a As shown, the method includes:

[0063] Step 110: Obtain an instruction to adjust a set face area in the original face image according to the reference face image.

[0064] In this embodiment, the original face image is composed of a plurality of original grids divided by face key points. The original face image refers to a picture with a face, which can be a selfie or photo taken by a user, a screenshot of a person in a video or live broadcast, etc. In order to facilitate the adjustment of the facial features or face shape and other set face areas, the original face image is divided into a plurality of original grids in advance according to the face key points identified in the original face image, so that the deformation adjustment of the face can be performed as a unit of the original grid.

[0065] After obtaining an instruction to adjust the set facial area in the original facial image according to the reference facial image, a specified deformation adjustment operation can be performed on the face in the original facial image according to the reference facial image indicated by the adjustment instruction, for example, thinning the face, enlarging the eyes, raising the nose, reducing the lips, etc., and according to the adjustment instruction, it can be determined what kind of deformation each original grid in the original facial image needs to undergo.

[0066] Optionally, before obtaining the instruction to adjust the set face area in the original face image according to the reference face image, it can also include: obtaining face image data and identifying face key points in the face image data; dividing the face image data into multiple grids according to face association points, and the vertices of the grids include at least the face key points; drawing the grids to obtain the original face image for display, and using the drawn grids as the original grids.

[0067] In this embodiment, face detection can be performed on face image data to identify key face points contained in the face image data, that is, to locate key areas of the face, including eyebrows, eyes, nose, mouth, facial contours, etc. Figure 1b As shown in the figure, the number of facial key points can be set according to the actual situation. For static image processing, the real-time requirement is low, and denser facial key points can be detected. For live broadcasting, the real-time requirement is high, and sparser facial key points can be detected.

[0068] In this embodiment, the grid represents a single drawable entity, and its vertices include at least facial key points, that is, the facial key points are used as at least part of the vertices of the grid, and the facial image data is gridded and divided into two or more grids, such as Figure 1c As shown. Meshing is to divide concave polygons or polygons with intersecting edges into convex polygons, such as triangles, so that they can be rendered by the rendering engine. Among them, the grids are ordered and can form a grid sequence. When the rendering engine is called, each grid will be drawn in order, so that the face image data with grids can be displayed on the screen.

[0069] Step 120: According to the instruction, respectively obtain an original area face key point set and a reference area face key point set matching the face area in the original face image and the reference face image.

[0070] In this embodiment, in order to perform a specified deformation adjustment operation on the face in the original face image according to the reference face image indicated by the instruction, it is necessary to first obtain a set of original area face key points and a set of reference area face key points that match the face area in the original face image and the reference face image respectively, and then find the correspondence between the original area face key points and the reference area face key points, and determine the deformation adjustment operation that needs to be performed on each original grid in the original face image.

[0071] Step 130: Determine, based on the reference regional face key point set, an adjusted regional face key point set that matches the original regional face key point set.

[0072] Taking into account that the original face image and the reference face image may be inconsistent in image size, face shape, facial features, etc., it is necessary to find the key point positions of the corresponding original area face key points after adjustments to different aspects based on the face key points of each reference area.

[0073] Optionally, determining an adjusted regional face key point set that matches the original regional face key point set based on a reference regional face key point set may include: determining target fixed points in the original face image and the reference face image respectively; calculating position vectors between each regional face key point in the reference regional face key point set and the original regional face key point set and the corresponding target fixed points respectively, and forming a plurality of position vector pairs; calculating adjusted position vectors corresponding to each regional face key point in the original regional face key point set based on a preset change force and each position vector pair; and determining an adjusted regional face key point set that matches the original regional face key point set based on the adjusted position vectors.

[0074] In this embodiment, in order to determine the correspondence between the original facial image and the reference facial image, it is necessary to first determine a target fixed point in the original facial image and the reference facial image as a reference position point respectively. The position of the target fixed point in the original facial image and the reference facial image is the same. The correspondence between the original facial image and the reference facial image is determined by the relative position relationship between the key facial points in the original facial image and the reference facial image and the corresponding target fixed point.

[0075] The position vectors between the facial key points in each region and the corresponding target fixed points in the reference region facial key point set and the original region facial key point set can be calculated respectively, and multiple position vector pairs can be formed. In each position vector pair, the reference region facial key points in one position vector correspond to the original region facial key points in another position vector. According to the preset change intensity and the corresponding relationship between the two, the adjusted position vector corresponding to the original region facial key points can be calculated, and then according to the position of the corresponding target fixed point, the adjusted region facial key points matching the original region facial key points can be determined.

[0076] Step 140: Adjust the original grid in the original face image according to the adjusted regional face key point set to generate an adjusted face image.

[0077] Optionally, adjusting the original grid in the original face image according to the set of facial key points in the adjusted area to generate the adjusted face image may include: establishing a blank image that matches the original face image; determining the grid deformation mode of each original grid in the original face image according to the set of facial key points in the adjusted area; dividing a plurality of target deformed grids corresponding to the original grid in the blank image according to the grid deformation mode; mapping each pixel point in the original grid to the corresponding target deformed grid according to the position correspondence between the original grid and the target deformed grid to obtain the adjusted face image.

[0078] In this embodiment, after obtaining an instruction to adjust the set facial area in the original facial image according to the reference facial image, a blank image with the same size as the original facial image can be established. Alternatively, after determining the adjusted area facial key point set that matches the original area facial key point set, a blank image can be established to facilitate the subsequent division of the blank image into a target deformed grid corresponding to the original grid, and after deforming and adjusting each original grid in the original face, the adjusted face can be displayed in the blank image to obtain an adjusted image.

[0079] Optionally, by comparing the face key point set of the adjusted area with the face key point set of the original area, the mesh deformation mode of each original mesh in the original face image can be determined. For example, the vertex c in the original mesh s is moved to the specified position point c', and the vertex d is moved to the specified position point d'. Then, in the blank image, according to the original mesh s and the corresponding mesh deformation mode, the target deformed mesh corresponding to the original mesh s can be divided.

[0080] In this embodiment, in order to speed up the processing of the original facial image, after dividing a plurality of target deformed grids corresponding to the original grid, the pixel points in each original grid are mapped to the corresponding target deformed grid in turn directly according to the mapping relationship between the original grid and the target deformed grid, without the need to re-render the pixel points in the original grid into the target deformed grid.

[0081] Optionally, based on the position correspondence between the original mesh and the target deformed mesh, each pixel point in the original mesh is mapped to the corresponding target deformed mesh to obtain an adjusted facial image, which may include: obtaining an original mesh in the original facial image as the current processing mesh; in a blank image, obtaining the target deformed mesh that matches the current processing mesh as a pairing mesh; obtaining a first vertex sequence corresponding to the current processing mesh, and a second vertex sequence corresponding to the pairing mesh, and calculating a mapping relationship matrix between the current processing mesh and the pairing mesh based on the first vertex sequence and the second vertex sequence; based on the mapping relationship matrix, mapping each pixel point in the current processing mesh to the pairing mesh, and returning to execute the operation of obtaining an original mesh in the original facial image as the current processing mesh, until all original meshes are processed.

[0082] In this embodiment, when mapping the pixel points in each original grid in the original face image, one of the multiple original grids can be selected as the current processing grid. For example, the original grid A can be selected from the target adjustment grid as the current processing grid, and then the target deformed grid a matching the current processing grid can be divided out from the blank image as the matching grid. Figure 1dAs shown. Then, the first vertex sequence (x1, y1), (x2, y2), (x3, y3) corresponding to the current processing grid A, that is, the coordinates of the three vertices of the current processing grid A, and the second vertex sequence (x1', y1'), (x2', y2'), (x3', y3') corresponding to the paired grid a are obtained, and then the mapping relationship matrix between the current processing grid A and the paired grid a is calculated according to the vertex coordinates of the two. According to the mapping relationship matrix, the coordinate values ​​of each pixel point in the current processing grid A in the paired grid a are obtained, so that each pixel point in the current processing grid A can be directly mapped to the paired grid a. Then, one of the remaining original grids is selected as the current processing grid, and the above process is repeated until all the original grids in the original face image are processed and the adjusted face image is obtained.

[0083] In an embodiment of the present invention, an instruction for adjusting a set face area in an original face image according to a reference face image is obtained; the original face image is composed of a plurality of original grids divided by face key points; according to the instruction, an original region face key point set and a reference region face key point set matching the face area are obtained in the original face image and the reference face image respectively; according to the reference region face key point set, an adjusted region face key point set matching the original region face key point set is determined; according to the adjusted region face key point set, the original grid in the original face image is adjusted to generate an adjusted face image, thereby solving the problems in the prior art of limiting the size and shape of the original face image and the adjusted face image and low face adjustment efficiency, not limiting the size and shape of the original face image and the adjusted face image, improving the deformation adjustment speed of the original face image, and being adaptable to real-time face transformation of videos.

[0084] Embodiment 2

[0085] Figure 2a is a flow chart of an image processing method in Embodiment 2 of the present invention. This embodiment can be combined with each optional solution in the above embodiments. Figure 2a , the method may include the following steps:

[0086] Step 210: Divide the original face image into a plurality of original grids according to the face key points.

[0087] Step 220: Obtain an instruction to adjust the set face area in the original face image according to the reference face image.

[0088] Step 230: According to the instruction, respectively obtain an original area face key point set and a reference area face key point set matching the face area in the original face image and the reference face image.

[0089] Step 240: Determine target fixed points in the original face image and the reference face image respectively.

[0090] Optionally, determining the target fixed point in the original face image and the reference face image respectively may include: determining the minimum enclosing rectangles corresponding to the original face image and the reference face image respectively based on the sets of facial key points corresponding to the original face image and the reference face image respectively; in each minimum enclosing rectangle, obtaining the corner point at the same position as the target fixed point.

[0091] In this embodiment, based on the sets of facial key points corresponding to the original facial image and the reference facial image, respectively, the minimum enclosing rectangle covering the original facial area and the reference facial area can be determined. In this embodiment, the minimum enclosing rectangle can be replaced by other polygons such as the minimum enclosing square, the minimum enclosing pentagon, the minimum enclosing circle, etc., and then in each minimum enclosing rectangle, the vertex at the same position is obtained as the target fixed point to determine the correspondence between the facial key points of each original area and the facial key points of each reference area.

[0092] Optionally, determining the minimum enclosing rectangles corresponding to the original facial image and the reference facial image respectively based on the facial key point sets corresponding to the original facial image and the reference facial image respectively may include: obtaining the maximum value points in four directions of the image coordinate system in the facial key point sets corresponding to the original facial image and the reference facial image respectively; determining the minimum enclosing rectangles corresponding to the original facial image and the reference facial image respectively based on the maximum value points in four directions of the image coordinate system corresponding to each facial key point set.

[0093] For example, Figure 2b As shown, taking the minimum bounding rectangle of the original face image as an example, assuming that there are n facial key points in the original face image, and the coordinates of the i-th facial key point are (xi, yi), the maximum value points in the four directions of the image coordinate system can be obtained according to the following method, that is, the coordinates of the four vertices of the minimum bounding rectangle of the original face image are determined:

[0094] A' point coordinates: (min(x1,x2,…,xn),max(y1,y2,…,yn)),

[0095] B' point coordinates: (min(x1,x2,…,xn),min(y1,y2,…,yn)),

[0096] Coordinates of point C': (max(x1,x2,…,xn),min(y1,y2,…,yn)),

[0097] D' point coordinates: (max(x1,x2,…,xn),max(y1,y2,…,yn)),

[0098] By connecting vertices A', B', C' and D' in sequence, the minimum enclosing rectangle of the original face image can be obtained.

[0099] Step 250: Calculate the position vectors between the face key points in each region and the corresponding target fixed points in the reference region face key point set and the original region face key point set, and form a plurality of position vector pairs.

[0100] Step 260: Calculate the adjusted position vectors corresponding to the regional face key points in the original regional face key point set according to the preset change strength and each position vector pair.

[0101] Optionally, calculating the adjusted position vectors corresponding to the regional face key points in the original regional face key point set based on a preset change force and each position vector pair, may include: calculating the scaling coefficient between the original face image and the reference face image based on the minimum enclosing rectangle corresponding to the original face image and the reference face image respectively; calculating the adjusted position vectors corresponding to the regional face key points in the original regional face key point set based on the preset change force, each position vector pair, and the scaling coefficient.

[0102] In this embodiment, the scaling factor may be the ratio of the diagonal length of the minimum bounding rectangle corresponding to the original face image to the diagonal length of the minimum bounding rectangle corresponding to the reference face image, or may be the area ratio of the minimum bounding rectangle, or may be determined in other ways. After determining the scaling factor between the original face image and the reference face image, the position vector of the original face image may be scaled to the position vector size of the reference face image, so that the face transformation is not limited by the image size, and then the final transformation processing vector is obtained by vector mixing and superposition.

[0103] Optionally, calculating the adjusted position vectors corresponding to each regional face key point in the original regional face key point set according to a preset change force, each position vector pair, and a scaling coefficient may include: calculating the adjusted position vector XnzAz of the regional face key point Xn' in the original regional face key point set according to the formula: σ*XnA+(1-σ)*Xn'A'*K=XnzAz, XnzAz corresponding to the nth position vector pair (XnA, Xn'A'); wherein XnA is the position vector between the regional face key point Xn in the reference regional face key point set and the target fixed point A in the reference face image, Xn'A' is the position vector between the regional face key point Xn' in the original regional face key point set and the target fixed point A' in the original face image, Xn matches Xn', σ is the change force, σ∈[0,1], and K is the scaling coefficient.

[0104] In this embodiment, if the position vectors corresponding to all regional face key points in the original regional face key point set are adjusted and deformed respectively according to the formula, the overall transformation of the face shape and facial features of the original face image is finally achieved. If only the position vectors corresponding to the key points of a certain facial feature in the original regional face key point set are adjusted and deformed according to the formula, the transformation effect of the facial features is finally achieved, for example, only the mouth shape is changed, etc. After obtaining the adjusted position vector XnzAz, the adjusted facial key point coordinates (Xz, Yz) can be obtained by coordinate conversion.

[0105] Step 270: Determine, based on the adjusted position vector, a set of adjusted regional face key points that matches the set of original regional face key points.

[0106] Step 280: Adjust the original grid in the original face image according to the adjusted regional face key point set to generate an adjusted face image.

[0107] In an embodiment of the present invention, an instruction for adjusting a set face area in an original face image according to a reference face image is obtained; the original face image is composed of a plurality of original grids divided by face key points; according to the instruction, an original region face key point set and a reference region face key point set matching the face area are obtained in the original face image and the reference face image respectively; according to the reference region face key point set, an adjusted region face key point set matching the original region face key point set is determined; according to the adjusted region face key point set, the original grid in the original face image is adjusted to generate an adjusted face image, thereby solving the problems in the prior art of limiting the size and shape of the original face image and the adjusted face image and low face adjustment efficiency, not limiting the size and shape of the original face image and the adjusted face image, improving the deformation adjustment speed of the original face image, and being adaptable to real-time face transformation of videos.

[0108] Embodiment 3

[0109] Figure 3 FIG. 1 is a schematic diagram of the structure of an image processing device in Embodiment 3 of the present invention. This embodiment is applicable to the situation of performing rapid real-time deformation adjustment on the face in the image. Figure 3 As shown, the image processing device includes: an adjustment instruction acquisition module 310 , a key point set acquisition module 320 , a determination module 330 and an adjustment module 340 .

[0110] The adjustment instruction acquisition module 310 is used to acquire an instruction to adjust a set face area in an original face image according to a reference face image; the original face image is composed of a plurality of original grids divided by face key points;

[0111] A key point set acquisition module 320 is used to acquire, according to the instruction, an original region face key point set and a reference region face key point set matching the face region in the original face image and the reference face image respectively;

[0112] A determination module 330, configured to determine, based on the reference regional face key point set, an adjusted regional face key point set that matches the original regional face key point set;

[0113] The adjustment module 340 is used to adjust the original grid in the original face image according to the face key point set in the adjusted area to generate an adjusted face image.

[0114] In an embodiment of the present invention, an instruction for adjusting a set face area in an original face image according to a reference face image is obtained; the original face image is composed of a plurality of original grids divided by face key points; according to the instruction, an original region face key point set and a reference region face key point set matching the face area are obtained in the original face image and the reference face image respectively; according to the reference region face key point set, an adjusted region face key point set matching the original region face key point set is determined; according to the adjusted region face key point set, the original grid in the original face image is adjusted to generate an adjusted face image, thereby solving the problems in the prior art of limiting the size and shape of the original face image and the adjusted face image and low face adjustment efficiency, not limiting the size and shape of the original face image and the adjusted face image, improving the deformation adjustment speed of the original face image, and being adaptable to real-time face transformation of videos.

[0115] Optionally, the determination module 330 is specifically used to: determine the target fixed points in the original face image and the reference face image respectively; calculate the position vectors between the reference area face key point set and the original area face key point set and the corresponding target fixed points, and form a plurality of position vector pairs; calculate the adjusted position vectors corresponding to the area face key points in the original area face key point set according to a preset change force and each position vector pair; determine the adjusted area face key point set that matches the original area face key point set according to the adjusted position vectors.

[0116] Optionally, the determination module 330 is specifically used to: determine the minimum enclosing rectangles corresponding to the original face image and the reference face image respectively based on the sets of facial key points corresponding to the original face image and the reference face image respectively; in each minimum enclosing rectangle, obtain the corner points at the same position as the target fixed points.

[0117] Optionally, the determination module 330 is specifically used to: obtain the maximum value points in the four directions of the image coordinate system in the facial key point sets corresponding to the original facial image and the reference facial image respectively; determine the minimum circumscribed rectangle corresponding to the original facial image and the reference facial image respectively based on the maximum value points in the four directions of the image coordinate system corresponding to each facial key point set.

[0118] Optionally, the determination module 330 is specifically used to: calculate the scaling coefficient between the original facial image and the reference facial image based on the minimum circumscribed rectangles corresponding to the original facial image and the reference facial image respectively; calculate the adjusted position vectors corresponding to the facial key points of each area in the original regional facial key point set based on a preset change force, each position vector pair, and the scaling coefficient.

[0119] Optionally, the determination module 330 is specifically used to: calculate the adjusted position vector XnzAz of the regional face key point Xn' in the original regional face key point set according to the formula: σ*XnA+(1-σ)*Xn'A'*K= XnzAz, XnzAz corresponds to the nth position vector pair (XnA, Xn'A'); wherein XnA is the position vector between the regional face key point Xn in the reference regional face key point set and the target fixed point A in the reference face image, Xn'A' is the position vector between the regional face key point Xn' in the original regional face key point set and the target fixed point A' in the original face image, Xn matches Xn', σ is the change intensity, σ∈[0,1], and K is the scaling coefficient.

[0120] Optionally, the adjustment module 340 is specifically used to: establish a blank image that matches the original facial image; determine the grid deformation mode of each original grid in the original facial image based on the set of facial key points in the adjusted area; divide a plurality of target deformed grids corresponding to the original grid in the blank image based on the grid deformation mode; map each pixel point in the original grid to the corresponding target deformed grid based on the position correspondence between the original grid and the target deformed grid to obtain an adjusted facial image.

[0121] Optionally, the adjustment module 340 is specifically used to: obtain an original mesh in the original face image as the current processing mesh; in the blank image, obtain a target deformed mesh that matches the current processing mesh as a pairing mesh; obtain a first vertex sequence corresponding to the current processing mesh, and a second vertex sequence corresponding to the pairing mesh, and calculate a mapping relationship matrix between the current processing mesh and the pairing mesh based on the first vertex sequence and the second vertex sequence; map each pixel point in the current processing mesh to the pairing mesh based on the mapping relationship matrix, and return to execute the operation of obtaining an original mesh in the original face image as the current processing mesh, until the processing of all the original meshes is completed.

[0122] Optionally, the adjustment instruction acquisition module 310 is also used to obtain facial image data and identify facial key points in the facial image data before obtaining an instruction to adjust the set facial area in the original facial image according to the reference facial image; divide the facial image data into multiple grids according to the facial association points, and the vertices of the grids include at least the facial key points; draw the grids to obtain the original facial image for display, and use the drawn grids as the original grids.

[0123] The image processing device provided by the embodiment of the present invention can execute the image processing method provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0124] Embodiment 4

[0125] Figure 4 It is a structural schematic diagram of a device disclosed in Embodiment 4 of the present invention. Figure 4 A block diagram of an exemplary device 12 suitable for use in implementing embodiments of the present invention is shown. Figure 4 The device 12 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0126] like Figure 4 As shown, device 12 is in the form of a general purpose computing device. Components of device 12 may include, but are not limited to, one or more processors or processing units 16, system memory 28, and bus 18 that connects various system components (including system memory 28 and processing unit 16).

[0127] Bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor or a local bus using any of a variety of bus architectures. By way of example, these architectures include, but are not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MAC) bus, an Enhanced ISA bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.

[0128] Device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by device 12, including volatile and non-volatile media, removable and non-removable media.

[0129] System memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. Device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 34 may be used to read and write non-removable, non-volatile magnetic media ( Figure 4 not shown, usually called a "hard drive"). Although Figure 4 Not shown in the figure, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, a DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to the bus 18 via one or more data medium interfaces. The memory 28 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of the embodiments of the present invention.

[0130] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in the memory 28, such program modules 42 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. The program modules 42 generally perform the functions and / or methods of the embodiments described herein.

[0131] The device 12 may also communicate with one or more external devices 14 (e.g., keyboards, pointing devices, displays 24, etc.), may communicate with one or more devices that enable a user to interact with the device 12, and / or may communicate with any device that enables the device 12 to communicate with one or more other computing devices (e.g., network cards, modems, etc.). Such communication may be performed via an input / output (I / O) interface 22. Furthermore, the device 12 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 20. As shown, the network adapter 20 communicates with other modules of the device 12 via a bus 18. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the device 12, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0132] The processing unit 16 executes various functional applications and data processing by running programs stored in the system memory 28, such as implementing the image processing method provided by the embodiment of the present invention.

[0133] That is: implement an image processing method, including:

[0134] Obtaining an instruction to adjust a set face area in an original face image according to a reference face image; The original face image is composed of a plurality of original grids divided by face key points;

[0135] According to the instruction, respectively obtaining a set of original area face key points and a set of reference area face key points matching the face area in the original face image and the reference face image;

[0136] Determine, based on the reference regional face key point set, an adjusted regional face key point set that matches the original regional face key point set;

[0137] According to the set of facial key points in the adjusted area, the original grid in the original face image is adjusted to generate an adjusted face image.

[0138] Embodiment 5

[0139] Embodiment 5 of the present invention further discloses a computer storage medium on which a computer program is stored. When the program is executed by a processor, an image processing method is implemented, including:

[0140] Obtaining an instruction to adjust a set face area in an original face image according to a reference face image; The original face image is composed of a plurality of original grids divided by face key points;

[0141] According to the instruction, respectively obtaining a set of original area face key points and a set of reference area face key points matching the face area in the original face image and the reference face image;

[0142] Determine, based on the reference regional face key point set, an adjusted regional face key point set that matches the original regional face key point set;

[0143] According to the set of facial key points in the adjusted area, the original grid in the original face image is adjusted to generate an adjusted face image.

[0144] The computer storage medium of the embodiment of the present invention can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, - but not limited to - an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0145] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0146] Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0147] Computer program code for performing the operations of the present invention may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0148] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. An image processing method, characterized in that: include: Obtaining an instruction to adjust a set face area in an original face image according to a reference face image; the original face image is composed of a plurality of original grids divided by face key points; According to the instruction, respectively obtaining an original area face key point set and a reference area face key point set matching the face area in the original face image and the reference face image; Determining, based on the reference regional face key point set, an adjusted regional face key point set that matches the original regional face key point set; According to the adjusted regional face key point set, adjusting the original grid in the original face image to generate an adjusted face image; The step of determining, based on the reference regional face key point set, a set of adjusted regional face key points that matches the original regional face key point set comprises: Determining target fixed points in the original face image and the reference face image respectively; Respectively calculating the position vectors between the face key points of each region and the corresponding target fixed points in the reference region face key point set and the original region face key point set, and forming a plurality of position vector pairs; Calculating the adjusted position vector corresponding to each regional face key point in the original regional face key point set according to the preset change strength and each of the position vector pairs; Determining, according to the adjusted position vector, a set of adjusted regional face key points that matches the set of original regional face key points; The step of calculating the adjusted position vector corresponding to each regional face key point in the original regional face key point set according to the preset change strength and each of the position vector pairs includes: Calculating a scaling factor between the original face image and the reference face image according to the minimum bounding rectangles corresponding to the original face image and the reference face image respectively; According to the preset change strength, each of the position vector pairs, and the scaling coefficient, the adjusted position vector corresponding to each regional face key point in the original regional face key point set is calculated.

2. The method according to claim 1, characterized in that Determining target fixed points in the original face image and the reference face image respectively includes: Determine the minimum circumscribed rectangles corresponding to the original face image and the reference face image, respectively, based on the sets of facial key points corresponding to the original face image and the reference face image, respectively; In each of the minimum circumscribed rectangles, a corner point at the same position is obtained as the target fixed point.

3. The method according to claim 2, characterized in that Determining the minimum bounding rectangles corresponding to the original face image and the reference face image respectively according to the sets of face key points corresponding to the original face image and the reference face image respectively, comprising: In the facial key point sets corresponding to the original facial image and the reference facial image, respectively, obtaining maximum value points in four directions of the image coordinate system; According to the maximum value points in four directions of the image coordinate system corresponding to each of the facial key point sets, the minimum circumscribed rectangles corresponding to the original facial image and the reference facial image are determined.

4. The method according to claim 1, characterized in that: Calculating the adjusted position vector corresponding to each regional face key point in the original regional face key point set according to the preset change strength, each position vector pair, and the scaling coefficient, including: According to the formula: σ*XnA+(1-σ)*Xn'A'*K=XnzAz, the adjusted position vector XnzAz of the regional face key point Xn' in the original regional face key point set is calculated, and XnzAz corresponds to the nth position vector pair (XnA, Xn'A'); Among them, XnA is the position vector between the regional face key point Xn in the reference regional face key point set and the target fixed point A in the reference face image, Xn'A' is the position vector between the regional face key point Xn' in the original regional face key point set and the target fixed point A' in the original face image, Xn matches Xn', σ is the change intensity, σ∈[0,1], and K is the scaling coefficient.

5. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: adjusting the original grid in the original face image according to the adjusted regional face key point set to generate an adjusted face image, including: Creating a blank image that matches the original face image; Determining a grid deformation mode of each of the original grids in the original face image according to the adjusted regional face key point set; According to the grid deformation mode, a plurality of target deformed grids corresponding to the original grid are divided in the blank image; According to the position correspondence between the original grid and the target deformed grid, each pixel point in the original grid is mapped to the corresponding target deformed grid to obtain the adjusted face image.

6. The method according to claim 5, characterized in that According to the position correspondence between the original grid and the target deformed grid, each pixel point in the original grid is mapped to the corresponding target deformed grid to obtain the adjusted face image, including: Acquire an original grid in the original face image as a current processing grid; In the blank image, obtaining a target deformed grid that matches the current processing grid as a paired grid; Acquire a first vertex sequence corresponding to the currently processed mesh and a second vertex sequence corresponding to the paired mesh, and calculate a mapping relationship matrix between the currently processed mesh and the paired mesh according to the first vertex sequence and the second vertex sequence; According to the mapping relationship matrix, each pixel point in the current processing grid is mapped to the paired grid, and the operation of obtaining an original grid in the original face image as the current processing grid is returned to be executed until the processing of all the original grids is completed.

7. The method according to any one of claims 1 to 4, characterized in that: Before obtaining the instruction to adjust the set face area in the original face image according to the reference face image, the method further includes: Acquire facial image data, and identify facial key points in the facial image data; According to the face association points, the face image data is divided into a plurality of grids, wherein the vertices of the grids at least include the face key points; The grids are drawn to obtain the original face image for display, and each of the drawn grids is used as the original grid.

8. An image processing device, characterized in that: include: An adjustment instruction acquisition module, used to acquire an instruction to adjust a set face area in an original face image according to a reference face image; the original face image is composed of a plurality of original grids divided by face key points; A key point set acquisition module, used to acquire, according to the instruction, an original region face key point set and a reference region face key point set matching the face region in the original face image and the reference face image respectively; A determination module, configured to determine, based on the reference regional face key point set, a set of adjusted regional face key points that matches the original regional face key point set; An adjustment module, configured to adjust the original grid in the original face image according to the adjusted regional face key point set, to generate an adjusted face image; The determination module is specifically used for: Determining target fixed points in the original face image and the reference face image respectively; Respectively calculating the position vectors between the face key points of each region and the corresponding target fixed points in the reference region face key point set and the original region face key point set, and forming a plurality of position vector pairs; Calculating the adjusted position vector corresponding to each regional face key point in the original regional face key point set according to the preset change strength and each of the position vector pairs; Determining, according to the adjusted position vector, a set of adjusted regional face key points that matches the set of original regional face key points; The step of calculating the adjusted position vector corresponding to each regional face key point in the original regional face key point set according to the preset change strength and each of the position vector pairs includes: Calculating a scaling factor between the original face image and the reference face image according to the minimum bounding rectangles corresponding to the original face image and the reference face image respectively; According to the preset change strength, each of the position vector pairs, and the scaling coefficient, the adjusted position vector corresponding to each regional face key point in the original regional face key point set is calculated.

9. A device, characterized in that: The device comprises: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the image processing method as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the image processing method as described in any one of claims 1 to 7 is implemented.

Images