Image processing method, device, equipment and computer readable storage medium

Abstract

Embodiments of the present application provide an image processing method, device and equipment and a computer readable storage medium, wherein the method comprises: obtaining a to-be-processed image and a plurality of feature points of the to-be-processed image; determining a target feature point to be moved and a target position of the target feature point based on a received feature point moving instruction; determining a target position of other feature points based on an original position of the target feature point, the target position, original positions of the other feature points and a preset attenuation function, the other feature points being each feature point except the target feature point; determining a first moving distance of each feature point, and determining a target texture coordinate of each feature point based on the first moving distance, the original position and a harmonic texture mapping algorithm; and rendering a processed image based on the target position of each feature point and the target texture coordinate of each feature point. Through the present application, a smooth and natural large-scale deformation effect can be achieved.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of image processing, and relate to but are not limited to an image processing method, device, equipment and computer readable storage medium. BACKGROUND

[0002] The research on face morphing technology has always been one of the important topics in image processing research. In short videos, face special effects such as face fusion, face changing, and some beautiful shapes in virtual images such as slim face and large eyes have wide applications. Real-time dragging morphing is difficult in face morphing technology due to the influence of large-scale deformation, the need for real-time speed, and the need to ensure the smoothness and naturalness of the deformation edge. Most slim face, large eye and other beauty algorithms mainly focus on small-scale deformation in a small local area, and cannot achieve smooth and natural large-scale deformation effect. SUMMARY

[0003] Embodiments of the present application provide an image processing method, device, equipment and computer readable storage medium, which realizes smooth and natural large-scale deformation effect through morphing position calculation based on vector synthesis of attenuation mechanism and texture coordinate calculation based on self-adaptation.

[0004] The technical solution of the embodiments of the present application is implemented as follows:

[0005] The embodiments of the present application provide an image processing method, comprising:

[0006] Obtaining a to-be-processed image, and performing feature extraction on the to-be-processed image to obtain a plurality of image feature points;

[0007] Performing interpolation processing on the plurality of image feature points to obtain a plurality of interpolated feature points;

[0008] Based on the received feature point movement instruction, determining a target feature point to be moved and a target position of the target feature point;

[0009] Based on the original position, target position of the target feature point, original positions of other feature points and a preset attenuation function, determining target positions of the other feature points, which are each feature point except the target feature point;

[0010] Determining a first movement distance of each feature point, and determining a target texture coordinate of each feature point based on the first movement distance, original position of each feature point and a harmonic texture mapping algorithm;

[0011] Rendering a processed image based on the target positions of the plurality of feature points and the target texture coordinates of the plurality of feature points.

[0012] The embodiments of the present application provide an image processing device, comprising:

[0013] The first acquisition module is configured to acquire a to-be-processed image, and perform feature extraction on the to-be-processed image to obtain a plurality of image feature points.

[0014] The interpolation processing module is configured to perform interpolation processing on the plurality of image feature points to obtain a plurality of interpolated feature points.

[0015] The first determination module is configured to determine a target feature point to be moved and a target position of the target feature point based on the received feature point movement instruction.

[0016] The second determination module is configured to determine target positions of other feature points based on the original position and the target position of the target feature point, the original positions of the other feature points, and a preset attenuation function, the other feature points being each feature point except the target feature point.

[0017] The third determination module is configured to determine first movement distances of the feature points, and determine target texture coordinates of the feature points based on the first movement distances, the original positions of the feature points, and a harmonic texture mapping algorithm.

[0018] The rendering module is configured to render a processed image based on the target positions of the feature points and the target texture coordinates of the feature points.

[0019] In some embodiments, the second determination module is further configured to:

[0020] determine, based on the original positions of the other feature points and the original positions of the target feature points, first distances between the other feature points and the target feature points;

[0021] determine, based on the first distances and a preset influence distance threshold, influence weights of the target feature points on the other feature points;

[0022] determine candidate target positions of the other feature points based on the original positions, the target positions, and the influence weights of the target feature points;

[0023] determine the target positions of the other feature points based on the original positions, the candidate target positions, and a preset attenuation function.

[0024] In some embodiments, the second determination module is further configured to:

[0025] when an ith first distance is less than or equal to the influence distance threshold, determine a ratio of the ith first distance to the influence distance threshold as a first parameter of an influence function; i = 1, 2, … N, N being a total number of target feature points.

[0026] determine an influence weight of the i-th target feature point on the other feature points based on the second parameter preset according to the influence function and the first parameter;

[0027] when the i-th first distance is greater than the influence distance threshold, determine a preset value as the influence weight of the i-th target feature point on the other feature points.

[0028] In some embodiments, the second determining module is further configured to:

[0029] determine a target vector of each target feature point based on an original position of the target feature point and a corresponding target position of the target feature point;

[0030] determine a candidate moving vector of the other feature points based on the influence weight of the target feature point on the other feature points and the target vector;

[0031] determine a candidate target position of the other feature points based on the candidate moving vector and the original position of the other feature points.

[0032] In some embodiments, the second determining module is further configured to:

[0033] determine a second moving distance of the other feature points based on the original position and the candidate target position of the other feature points;

[0034] when the second moving distance is greater than a preset adjustment distance threshold, determine an attenuation value of the other feature points based on the second moving distance and an attenuation function;

[0035] determine a target position of the other feature points based on the original position, the candidate target position, the adjustment distance threshold and the attenuation value of the other feature points.

[0036] In some embodiments, the second determining module is further configured to:

[0037] when the second moving distance is less than or equal to the adjustment distance threshold, determine the candidate target position as the target position of the other feature points.

[0038] In some embodiments, the second determining module is further configured to:

[0039] determine a unit vector of the other feature points in a moving direction based on the original position and the candidate target position of the other feature points;

[0040] determine a third parameter beyond the adjustment distance threshold based on the second moving distance, the adjustment distance threshold and the attenuation value, and determine the adjustment distance threshold as a fourth parameter;

[0041] determine a target movement vector of the other feature point based on the third parameter, the fourth parameter and the unit vector;

[0042] determine a target position of the other feature point based on the target movement vector and an original position of the other feature point.

[0043] In some embodiments, the interpolation processing module is further configured to:

[0044] interpolate the extracted image feature points to obtain interpolated image feature points;

[0045] determine contour feature points of a target image region based on the interpolated image feature points, and acquire image edge feature points of the image to be processed;

[0046] determine the interpolated image feature points, the contour feature points and the image edge feature points as a plurality of interpolated feature points;

[0047] In some embodiments, the apparatus further comprises:

[0048] a triangulation module configured to triangulate a plurality of feature points of the image to be processed to obtain index information of a plurality of triangular patches.

[0049] In some embodiments, the third determination module is further configured to:

[0050] determine an original texture coordinate of the qth feature point based on an original position of the qth feature point and size information of the image to be processed;

[0051] when the first movement distance of the qth feature point is less than or equal to the adjustment distance threshold, determine the original texture coordinate of the qth feature point as a target texture coordinate of the qth feature point.

[0052] In some embodiments, the third determination module is further configured to:

[0053] when the first movement distance of the qth feature point is greater than the adjustment distance threshold, re-perform triangulation based on the target positions of the feature points to obtain index information of a plurality of triangular patches;

[0054] determine a target texture coordinate of the qth feature point based on the index information of the plurality of triangular patches and a harmonic mapping algorithm.

[0055] Embodiments of the present application provide an image processing device, comprising:

[0056] a memory configured to store executable instructions, and a processor configured to execute the executable instructions stored in the memory to implement the method described above.

[0057] The embodiment of the present application provides a computer readable storage medium, which stores executable instructions, and when the executable instructions are executed by a processor, the method described above is realized.

[0058] The embodiment of the present application has the following beneficial effects:

[0059] In the embodiment of the present application, after the to-be-processed image is acquired and the plurality of feature points of the face image are acquired, the user can trigger a moving instruction of the feature point movement, at this time, based on the received feature point moving instruction, the target feature point to be moved and the target position of the target feature point are determined, and the movement of the target feature point acts on other feature points except the target feature point, the target position of the other feature points can be determined based on the original position of the target feature point, the target position, the original position of the other feature points and a preset attenuation function, and the target texture coordinates of each feature point are determined based on the first moving distance of each feature point, the original position and the harmonic texture mapping algorithm, and finally, the processed image is rendered based on the target position of each feature point and the target texture coordinates of each feature point, and the attenuation function is used to attenuate the influence of the target feature point on the other feature points when the moving distance of the other feature points under the influence of the target feature point exceeds a preset influence distance threshold, so as to attenuate the moving distance of the other feature points, so that a smooth and natural effect can be achieved in large-scale deformation. BRIEF DESCRIPTION OF DRAWINGS

[0060] Figure 1 is a network architecture schematic diagram of an image processing system 10 provided by the embodiment of the present application;

[0061] Figure 2 is a structural schematic diagram of a terminal 300 provided by the embodiment of the present application;

[0062] Figure 3 is an implementation flow schematic diagram of an image processing method provided by the embodiment of the present application;

[0063] Figure 4 is an implementation flow schematic diagram of determining the target position of the other feature points provided by the embodiment of the present application;

[0064] Figure 5 is another implementation flow schematic diagram of an image processing method provided by the embodiment of the present application;

[0065] Figure 6 is another implementation flow schematic diagram of an image processing method provided by the embodiment of the present application;

[0066] Figure 7 is a schematic diagram of the extracted face feature points provided by the embodiment of the present application;

[0067] Figure 8An image diagram obtained by triangulating the interpolated feature points is provided for the embodiments of the present application.

[0068] Figure 9 An effect diagram of the constraint points on the free points is provided for the embodiments of the present application.

[0069] Figure 10 A comparative diagram for large-scale deformation of an image is provided. DETAILED DESCRIPTION

[0070] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings, and the described embodiments should not be regarded as limiting the present application. All other embodiments obtained by those of ordinary skill in the art without making creative efforts fall within the scope of protection of the present application.

[0071] In the following description, "some embodiments" are referred to, which describe a subset of all possible embodiments, but it can be understood that "some embodiments" can be the same subset or different subsets of all possible embodiments, and can be combined with each other without conflict. Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meanings as understood by those skilled in the art to which the embodiments of the present application belong. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0072] The embodiments of the present application provide a relatively complete and robust real-time large-scale drag deformation scheme. First, a face feature recognition algorithm is used to extract facial feature points, and forehead points, cheek points, points on the outer edge of the contour, and image boundary points are interpolated on the original feature points, which expands the deformation area and is beneficial to dividing the entire image. Some constraint points are set from the interpolated feature points, which are arbitrarily dragged by the user, and a vector synthesis deformation position calculation and adaptive texture coordinate calculation scheme based on a decay mechanism is provided in the embodiments of the present application. A smooth and natural large-scale deformation effect is obtained in real time through OpenGL rendering, and the problems of speed, large scale, and smooth and natural deformation are solved.

[0073] The exemplary application of the image processing device provided by the embodiments of the present application will be described below. The image processing device provided by the embodiments of the present application can be implemented as any terminal having a screen display function, such as a notebook computer, a tablet computer, a desktop computer, a mobile device (for example, a mobile phone, a portable music player, a personal digital assistant, a dedicated messaging device, a portable game device), a smart robot, and the like, or as a server. Below, the exemplary application of the image processing device implemented as a terminal will be described.

[0074] Referring to Figure 1 ,Figure 1 is a network architecture schematic diagram of the image processing system 10 provided by an embodiment of the present application. As shown in the figure, the network architecture includes a server 100, a network 200, and a terminal 300, wherein the network 200 can be a wide area network or a local area network, or a combination of the two. The server 100 can be a stand-alone physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDNs, and basic cloud computing services such as big data and artificial intelligence platforms. The above is only an example, and the present application is not limited in this regard. Figure 1

[0075] The terminal 300 runs an application program, which can be an instant messaging application program, a shopping application program, and an image acquisition application program. The image processing method provided by the present application can be implemented through a dedicated application program, such as a beauty application program, or embedded in other application programs in the form of a functional plug-in, such as an instant messaging application program or a video watching application program. In implementing the image processing method of the present application, the terminal 300 first acquires a to-be-processed image, which can be acquired by the terminal 300 using an image acquisition application program, downloaded from a network, or sent by another terminal. The terminal 300 first acquires feature points of the to-be-processed image and performs triangular partitioning. When the terminal 300 receives a movement operation on some of the feature points, it determines the target positions of these feature points, and determines the target positions of other feature points based on the target positions of these feature points based on the attenuation mechanism, and adaptively performs texture coordinate calculation to obtain a processed image and render the processed image. Then, the terminal 300 sends the processed image to the server 100 based on the received image upload instruction.

[0076] Referring to Figure 2 , Figure 2 is a structural schematic diagram of the terminal 300 provided by an embodiment of the present application, Figure 2 The terminal 300 shown in the figure includes at least one processor 310, a memory 350, at least one network interface 320, and a user interface 330. The various components in the terminal 300 are coupled together through a bus system 340. It can be understood that the bus system 340 is used to realize the connection and communication between the components. In addition to including a data bus, the bus system 340 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, all kinds of buses are marked as the bus system 340 in the figure. Figure 2

[0077] ​​The processor 310 can be an integrated circuit chip having a processing capability of signals, such as a general purpose processor, a digital signal processor (DSP), or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, etc., wherein the general purpose processor can be a microprocessor or any conventional processor.

[0078] The user interface 330 includes one or more output devices 331 enabling presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 330 also includes one or more input devices 332, including user interface components facilitating user input, such as a keyboard, a mouse, a microphone, a touch screen display, a camera, other input buttons and controls.

[0079] The memory 350 can be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard drives, optical drives, etc. The memory 350 optionally includes one or more storage devices remotely located from the processor 310. The memory 350 comprises volatile memory or non-volatile memory, and can also include both volatile and non-volatile memory. Non-volatile memory can be read only memory (ROM), volatile memory can be random access memory (RAM). The memory 350 described in embodiments of the present application is intended to include any suitable type of memory. In some embodiments, the memory 350 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or a subset or superset thereof, which are described below.

[0080] The operating system 351 includes a system program for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0081] The network communication module 352 is used to reach other computing devices via one or more (wired or wireless) network interfaces 320, exemplary network interfaces 320 including Bluetooth, wireless compatibility authentication (WiFi), and universal serial bus (USB), etc.

[0082] The input processing module 353 is used to detect and translate one or more user inputs or interactions from one or more input devices 332.

[0083] In some embodiments, the apparatus provided in this application can be implemented in software. Figure 2 An image processing device 354 stored in memory 350 is shown. This image processing device 354 can be an image processing device in terminal 300, and can be software in the form of programs and plug-ins, including the following software modules: a first acquisition module 3541, an interpolation processing module 3542, a first determination module 3543, a second determination module 3544, a third determination module 3545, and a rendering module 3546. These modules are logically connected and can therefore be arbitrarily combined or further divided according to the functions they implement. The functions of each module will be described below.

[0084] In other embodiments, the apparatus provided in this application can be implemented in hardware. As an example, the apparatus provided in this application can be a processor in the form of a hardware decoding processor, which is programmed to execute the image processing method provided in this application. For example, the processor in the form of a hardware decoding processor can be one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0085] The image processing method provided in this application will be described below with reference to exemplary applications and implementations of the terminal 300 provided in the embodiments of this application. See also Figure 3 , Figure 3 This is a schematic diagram illustrating an implementation flow of the image processing method provided in this application embodiment, which will be combined with... Figure 3 The steps shown are explained.

[0086] Step S101: Obtain the image to be processed and perform feature extraction to obtain multiple image feature points.

[0087] Here, the image to be processed can be a real person image, a cartoon character image, an animal image, etc. The image to be processed includes all or part of the limb images of the target object, and the all or part of the limb images include at least the facial image area.

[0088] The image to be processed can be an image captured by the terminal using its image acquisition device (e.g., a camera), can be a previously stored image obtained from the storage space of the terminal, or can be an image downloaded from a network.

[0089] The feature extraction not only extracts features conducive to recognition from the original image information, but also greatly reduces the dimension of the image data. In step S101, the feature extraction of the image to be processed refers to the extraction of features of the body image region in the image to be processed, such as face feature extraction, leg feature extraction, arm feature extraction, neck feature extraction, etc. Further, a predetermined feature extraction algorithm can be used to extract features of the image to be processed, such as the HOG algorithm, the Dlib algorithm, and in some embodiments, a neural network model can be used to extract feature points of the image to be processed.

[0090] In step S102, the plurality of image feature points are subjected to interpolation processing to obtain a plurality of interpolated feature points.

[0091] Here, the image feature points extracted in step S101 are relatively sparse and some feature points in some regions can be missing, such as the feature points in the forehead region when face feature extraction is performed. Therefore, in order to make the deformation smoother, the image feature points extracted need to be subjected to interpolation processing in step S102. In implementation, the sparse image feature points extracted can be subjected to interpolation processing first, and then the feature points on the outer contour of the body region and the edge feature points of the image to be processed are interpolated.

[0092] In some embodiments, after the plurality of feature points are obtained, the image to be processed is subjected to triangular subdivision based on the plurality of feature points to obtain a plurality of triangular patches.

[0093] In step S103, based on the received feature point movement instruction, a target feature point to be moved and a target position of the target feature point are determined.

[0094] In the embodiments of the present application, the body image region of the target object (e.g., a real person, a cartoon character, or an animal) is deformed, so the target feature point to be moved is in the body image region, such as a target feature point in the face region, a target feature point in the leg region, a target feature point in the arm region, a target feature point in the neck region, etc. The feature point movement instruction can be triggered based on a drag operation of a user on at least one feature point. At this time, the target feature point to be moved can be determined according to the action position of the drag operation, and the target position of the target feature point can be determined according to the termination position of the drag operation. In actual implementation, the drag operation can be an operation on one feature point, or can be an operation on a plurality of feature points.

[0095] It should be noted that when the feature point moving instruction is triggered by the drag operation, the target feature point to be moved can be determined directly according to the feature point corresponding to the action position of the drag operation. Since the user generally drags the facial feature points, and the face is symmetrical, in order to simplify the user operation and to avoid the problem that the left and right distances are inconsistent when the user drags left and right, when the target feature point to be moved is determined according to the action position of the drag operation, the feature point corresponding to the action position of the drag operation and the feature point symmetrical to the feature point can be determined as the target feature point to be moved at the same time. For example, the feature point corresponding to the action position of the drag operation is on the left cheek, and then the feature point on the left cheek and the feature point on the right cheek symmetrical to the feature point are determined as the target feature point.

[0096] In some embodiments, a human-computer interaction interface for setting the movement of the feature points can be provided, and the user can select one or more target feature points to be moved through the human-computer interaction interface, and can also set the movement distance, movement direction, etc. of the target feature points to be moved. At this time, the feature point moving instruction can be triggered after setting the setting information of the feature point identifier, movement distance, movement direction, etc. of the target feature points to be moved. At this time, the target feature points to be moved can be determined based on the feature point identifier in the setting information of the user, and the target position of the target feature point can be determined based on the movement distance, movement direction, and initial position of the target feature point.

[0097] In step S104, the target positions of the other feature points are determined based on the original positions and target positions of the target feature points, the original positions of the other feature points, and a preset attenuation function.

[0098] Here, the other feature points are feature points other than the target feature points, and the attenuation function is used to attenuate the influence of the target feature points on the other feature points when the movement distance of the other feature points under the influence of the target feature points exceeds a preset influence distance threshold, so as to reduce the movement speed of the other feature points following the target feature points.

[0099] In the implementation of step S104, the distance between a certain other feature point and each target feature point can be calculated first, the influence weight of each target feature point on the other feature point is determined according to the distance, and then the target position of the other feature point is determined according to the influence weight and the movement vector of each target feature point.

[0100] In some embodiments, when the distance between the target feature point and other feature points is greater than the preset influence distance threshold, the influence weight between the target feature point and other feature points can be a preset value, for example, can be 0, that is, when the distance between the target feature point and a certain other feature point is greater than the preset influence distance threshold, the influence of the target feature point on the other feature point is 0, that is, the other feature point will not move because of the movement of the target feature point. Therefore, when determining the target position of the other feature points, the calculation is for those other feature points whose distance with the target feature point is less than or equal to the influence distance threshold, and the position of the other feature points whose distance with each target feature point is greater than the influence distance threshold will not change, at this time the original position is the target position.

[0101] Step S105, determine the first movement distance of each feature point, and determine the target texture coordinate of each feature point based on the first movement distance of each feature point, the original position and the harmonic texture mapping algorithm.

[0102] Here, since the feature point movement instruction can make the target feature point move to a position far away from the original position, and in the embodiment of the present application, the influence distance threshold is set, so that other feature points following the movement of the target feature point move within their own influence domain, so the texture coordinates of the target feature point and other feature points will be different, when the movement distance of the target feature point exceeds the influence domain, the texture coordinates should be determined by the texture coordinates of the target feature point near the target position, and the texture coordinates of the other feature points within the influence domain remain unchanged, so that the stretching of the texture generates deformation. Here, for the target feature point, the grid topology can be used to obtain the texture coordinates with the minimum texture space deformation by referring to the principle of two-dimensional harmonic mapping.

[0103] In the implementation of step S105, first, the original texture coordinates of each feature point are determined, then when the first movement distance of a certain feature point is less than or equal to the preset influence distance threshold, the texture coordinates of the feature point remain unchanged, that is, the target texture coordinates are the original texture coordinates; when the first movement distance of the feature point is greater than the influence distance threshold, the feature points after movement are re-triangulated, and the target texture coordinates of the feature point are determined by using the harmonic mapping algorithm.

[0104] Step S106, render the processed image based on the target positions of the feature points and the target texture coordinates of the feature points.

[0105] Here, after the target positions of the feature points and the corresponding target texture coordinates are determined, the processed image can be obtained, and then a rendering algorithm is used to render the processed image, for example, an OpenGL algorithm can be used to render the processed image.

[0106] In the embodiment of the present application, after the to-be-processed image is acquired and the plurality of feature points of the face image are acquired, a user can trigger a movement instruction of feature point movement, at this time, based on the received feature point movement instruction, a target feature point to be moved and a target position of the target feature point are determined, and the movement of the target feature point acts on other feature points except the target feature point, the target positions of the other feature points can be determined based on the original position of the target feature point, the target position, the original positions of the other feature points and a preset attenuation function, and the target texture coordinates of each feature point are determined based on the first movement distance of each feature point, the original position and the harmonic texture mapping algorithm, and finally a processed image is rendered based on the target positions of the feature points and the target texture coordinates of each feature point, and the attenuation function is used to attenuate the influence of the target feature point on the other feature points when the movement distance of the other feature points under the influence of the target feature point exceeds a preset influence distance threshold, so as to attenuate the movement distance of the other feature points, so that a smooth and natural effect can be achieved in large-scale deformation.

[0107] In some embodiments, Figure 3 The "interpolation processing is performed on the plurality of image feature points to obtain a plurality of interpolated feature points" in step S102 shown can be implemented by the following steps:

[0108] Step S1021, performing interpolation processing on the extracted image feature points to obtain interpolated image feature points;

[0109] In step S101, the extracted image feature points are relatively sparse. Taking a face image as an example, 68 feature points can be extracted, as shown in Figure 7 As shown in Figure 7 The 68 feature points do not include the feature points of the forehead edge, so step S1021 can be implemented by interpolating the middle forehead point and the top forehead point based on the 28th point and the 31st point on the nasal ala midline, and then interpolating the other forehead edge points in the forehead region according to the middle forehead point and the top forehead point. Figure 7 It can also be seen that the feature points of the cheek edge are also relatively sparse, so interpolation can be performed between each of the extracted cheek feature points to obtain the interpolated facial feature points.

[0110] Step S1022, determining the contour feature points of the target image region based on the interpolated image feature points, and acquiring the image edge feature points of the to-be-processed image.

[0111] Here, the contour feature points of the target image region can be points on a contour outside the target image region, that is, points on a contour outside the image region of the body in the image to be processed. Taking the image region of a human face as an example, in implementation, step S1022 can be to select feature points on the cheek edge and feature points on the forehead edge from the interpolated facial feature points, and then determine points corresponding to the feature points on the cheek edge and the feature points on the forehead edge after being moved outward by a certain distance as the contour feature points of the image region of the face. The image edge feature points of the image to be processed can be obtained by sampling each pixel point on the image edge.

[0112] Step S1023: determining the interpolated image feature points, the contour feature points, and the image edge feature points as the plurality of feature points after interpolation.

[0113] Correspondingly, after step S101, the method further includes:

[0114] Step S001: performing triangulation on the image to be processed based on the plurality of feature points after interpolation to obtain index information of a plurality of triangular facets.

[0115] Here, the plurality of feature points obtained in step S1023 constitute a feature point set of the image to be processed, and each feature point has its own number.

[0116] In implementation, step S001 can be to perform triangulation on the image to be processed based on the feature point set by using a Delaunay triangulation algorithm, so as to obtain a plurality of triangular facets with each feature point in the feature point set as a vertex, and to satisfy that, except for end points, edges of the plurality of triangular facets do not contain any point in the point set; the plurality of triangular facets do not have intersecting edges; all faces in the planar graph are triangular faces, and the union of all the triangular faces is a convex hull of the feature point set. In addition, the index information of each triangular facet, that is, the numbers of the vertex feature points of the triangular facet, for example, the vertexes of a triangular facet are the 6th feature point, the 79th feature point, and the 60th feature point, and the index information of the triangular facet is (6, 79, 60).

[0117] In the embodiments of steps S1021 to S1023 described above, the plurality of feature points of the image to be processed include not only the image feature points but also the image edge feature points and the contour feature points of the target image region. The image edge feature points can be used to perform triangulation on the entire image, and the contour feature points of the image region of the face can be used to make the division of the face contour and the boundary of the image more fine, so as to make the subsequent deformation more smooth.

[0118] In some embodiments, Figure 3The step S104 shown as "determining the target position of the other feature point based on the original position of the target feature point, the target position, the original position of the other feature point, and a preset attenuation function" can be implemented by the following steps:

[0119] In step S1041, the first distance between the other feature point and each target feature point is determined based on the original position of the other feature point and the original position of each target feature point.

[0120] Here, it is assumed that there are N target feature points, and the distance between a certain other feature point and the i-th target feature point is the i-th first distance.

[0121] In step S1041, the i-th first distance between the j-th other feature point P j and the i-th target feature point N i can be determined according to formula (1-1):

[0122]

[0123] wherein, is the original position of P j , and is the original position of N i .

[0124] In step S1042, the influence weight of each target feature point on the other feature point is determined based on the first distance and a preset influence distance threshold.

[0125] Here, the influence distance threshold is preset and used to determine the influence weight of each target feature point on the other feature point. In step S1042, when the first distance is less than or equal to the influence distance threshold, the influence weight of the target feature point on the other feature point is determined based on a preset influence function and the first distance, and when the first distance is greater than the influence distance threshold, a preset value can be directly determined as the influence weight of the target feature point on the other feature point, which can be 0 or approximately 0.

[0126] In step S1043, the candidate target position of the other feature point is determined based on the original position, the target position of each target feature point, and the influence weight.

[0127] Here, in step S1043, the moving vector of each target feature point can be determined first, and then the weighted sum of the moving vector of each target feature point and the corresponding influence weight can be determined to obtain the candidate moving vector of the other feature point, and then the candidate moving vector and the original position of the other feature point can be used to determine the candidate target position of the other feature point.

[0128] Step S1044, determining the target position of the other feature point based on the original position of the other feature point, the candidate target position and the preset attenuation function.

[0129] Here, since the candidate target position is directly determined based on the influence weight of the target feature point on the other feature point, the influence of the target feature point on the other feature point after moving a certain distance is not considered. Therefore, in step S1044, the moving distance of the other feature point is determined based on the original position of the other feature point and the candidate target position, and whether the candidate target position needs to be adjusted is determined based on the moving distance. When it is determined that the candidate target position needs to be adjusted, the moving vector of the other feature point is determined again based on the attenuation function and the moving position, and the final target position of the other feature point is determined.

[0130] Through the above steps S1041 to S1044, the influence weight of each target feature point on the other feature point can be determined according to the distance between the other feature point and each target feature point. The candidate moving vector of the other feature point is determined by performing weighted summation and vector synthesis based on the moving vector of each target feature point and the corresponding influence weight. The candidate target position of the other feature point is determined, and the target position of the other feature point is further determined based on the moving distance of the other feature point and the attenuation function. Therefore, after the other feature point moves a certain distance, the influence of the target feature point on the other feature point is attenuated. This can achieve smooth deformation and also adaptively solve the problem of large-scale deformation.

[0131] In some embodiments, the above step S1042 can be implemented by the following steps:

[0132] Step S421, determining whether the ith first distance is less than or equal to the influence distance threshold.

[0133] Here, when the ith first distance is less than or equal to the influence distance threshold, it means that the first distance between the other feature point and the ith target feature point is still within the preset influence domain range, and step S422 is entered; when the ith first distance is greater than the influence distance threshold, it means that the first distance between the other feature point and the ith target feature point has exceeded the influence domain range, and step S424 is entered.

[0134] Step S422, determining the ratio of the ith first distance to the influence distance threshold as the first parameter of the influence function.

[0135] Here, i = 1, 2, … N, and N is the total number of target feature points. In the embodiments of the present application, the influence function can be a Gaussian function as shown in formula (1-2):

[0136]

[0137] wherein r is a first parameter, i.e. d is the ith first distance, δ is an influence distance threshold, and r≤1.

[0138] Step S423, determining an influence weight of the ith target feature point on the other feature point based on the second parameter preset by the influence function and the first parameter.

[0139] Here, β in formula (1-2) is the preset second parameter. After the first parameter is determined, the preset second parameter and the first parameter are substituted into formula (1-2) to determine the influence weight of the ith target feature point on the other feature point.

[0140] Step S424, determining a preset value as the influence weight of the ith target feature point on the other feature point.

[0141] Here, the preset value is a small value preset in advance. For example, in actual implementation, the preset value can be 0 or a value close to 0, to indicate that when the first distance between the other feature point and the ith target feature point is greater than the influence distance threshold, it is determined that the influence of the target feature point on the other feature point is close to 0, i.e., the ith target feature point no longer has an influence on the other feature point.

[0142] In some embodiments, the above step S1043 “determining a candidate target position of the other feature point based on the original position, the target position, and the influence weight of each target feature point” can be implemented by the following steps:

[0143] Step S431, determining a target vector corresponding to each target feature point based on the original position and the target position of the target feature point.

[0144] Here, the target vector corresponding to the ith target feature point can be determined by formula (1-3):

[0145]

[0146] wherein, is the original position of the N i th target feature point, is the target position of the N i th target feature point.

[0147] Step S432, determining a candidate moving vector corresponding to the other feature point based on the influence weight of each target feature point on the other feature point and the target vector.

[0148] Here, step S432, when implemented, can be weighting and summing the influence weight of each target feature point on other feature points and each target vector, so as to obtain the candidate moving vector corresponding to the other feature point.

[0149] Step S433, determining the candidate target position of the other feature point based on the candidate moving vector and the original position of the other feature point.

[0150] Here, assuming that the candidate moving vector of the jth other feature point P j is and the original position of P j is P j , then the candidate target position of P j can be determined by formula (1-4):

[0151]

[0152] Through steps S431 to S433, the target vector of each target feature point and the corresponding influence weight can be weighted and summed to realize vector synthesis, so as to achieve faster calculation speed, and further determine the candidate moving vector of the other feature point, and then determine the target position of the other feature point according to the original position and the candidate moving vector of the other feature point, so as to realize large-scale smooth deformation.

[0153] In some embodiments, the above step S1044“determining the target position of the other feature point based on the original position, the candidate target position and the preset attenuation function of the other feature point” can be implemented by steps S441 to S445 as shown in the following. Figure 4 Each step will be described below. Figure 4

[0154] Step S441, determining the second moving distance of the other feature point based on the original position and the candidate target position of the other feature point.

[0155] Step S442, determining whether the second moving distance is greater than a preset adjustment distance threshold.

[0156] Here, when the second moving distance is greater than the adjustment distance threshold, it means that the candidate target position needs to be adjusted, and then step S443 is entered; when the second moving distance is less than or equal to the adjustment distance threshold, it means that the candidate target position does not need to be adjusted, and then step S445 is entered.

[0157] Step S443, determining the attenuation value of the other feature point based on the second moving distance and the attenuation function.

[0158] Here, in the embodiments of the present application, the attenuation function can be as shown in formula (1-5):

[0159]

[0160] Where σ is the attenuation parameter.

[0161] When determining s, the movement can be extended along the direction of other feature points until the edge of the image to be processed is reached, and the edge point G that intersects with the edge of the image to be processed in this movement direction is determined. j The location information is used to determine the original location of other feature points and the second distance between the edge points, and the ratio between the second movement distance and the second distance is determined as s, i.e., formula (1-6):

[0162]

[0163] Among them, The second distance between the original position of the other feature point and the edge point is l, where l is the second movement distance.

[0164] Step S444: Determine the target position of the other feature points based on their original positions, candidate target positions, adjusted distance threshold, and attenuation value.

[0165] In some embodiments, Figure 4 Step S444 shown can be achieved through the following steps:

[0166] Step S4441: Determine the unit vector of the other feature points in the direction of movement based on the original positions of the other feature points and the candidate target positions;

[0167] Here, the unit vector can be determined by formula (1-7):

[0168]

[0169] Step S4442: Based on the second moving distance, the adjusted distance threshold, and the attenuation value, determine a third parameter other than the adjusted distance threshold, and determine the adjusted distance threshold as the fourth parameter;

[0170] Here, the third parameter can be determined using formula (1-8):

[0171]

[0172] in, R is the attenuation value corresponding to the other feature point, and R is the adjusted distance threshold.

[0173] Step S4443: Based on the third parameter, the fourth parameter, and the unit vector, determine the target movement vector of the other feature points.

[0174] Here, the target movement vector can be determined by formula (1-9):

[0175]

[0176] Step S4444, determining the target position of the other feature point based on the target movement vector and the original position of the other feature point.

[0177] Here, when the target movement vector and the original position of the other feature point are determined, the original position coordinate and the target movement vector are added, and the final target position of the other feature point is obtained.

[0178] Step S445, determining the candidate target position as the target position of the other feature point.

[0179] Here, when the second movement distance is less than or equal to the adjustment distance threshold, it indicates that the other feature point is still within the adjustment domain, and the candidate target position is directly determined as the target position of the other feature point.

[0180] In some embodiments, the above step S105 "determining the target texture coordinate of each feature point based on the first movement distance, the original position of each feature point, and the harmonic texture mapping algorithm" can be implemented by the following steps:

[0181] Step S1051, determining the original texture coordinate of the qth feature point based on the original position of the qth feature point and the size information of the to-be-processed image.

[0182] In the embodiments of the present application, the to-be-processed image is a two-dimensional image, and the original texture coordinate of the qth feature point can be determined by formula (1-10):

[0183]

[0184] Wherein, width is the width of the to-be-processed image, and height is the height of the to-be-processed image.

[0185] Step S1052, determining whether the first movement distance of the qth feature point is less than or equal to the influence distance threshold.

[0186] Here, when the first movement distance of the qth feature point is less than or equal to the influence distance threshold, it indicates that the movement of the qth feature point is within the influence domain, and step S1053 is entered, and when the first movement distance of the qth feature point is greater than the influence distance threshold, it indicates that the qth feature point has moved out of the influence domain, and step S1054 is entered.

[0187] Step S1053, when the first movement distance of the qth feature point is less than or equal to the influence distance threshold, the original texture coordinate of the qth feature point is determined as the target texture coordinate of the qth feature point.

[0188] Step S1054, when the first movement distance of the qth feature point is greater than the adjustment distance threshold, based on the target positions of the respective feature points, re-triangulation is performed to obtain index information of a plurality of triangular patches.

[0189] Here, due to the movement of the target feature point, at least part of the other feature points will follow the movement, and when the movement distance is large, the adjacent points of each feature point will also change, so re-triangulation is performed at this time to obtain new index information of a plurality of triangular patches.

[0190] Step S1055, based on the index information of the plurality of triangular patches and the harmonic mapping algorithm, the target texture coordinate of the qth feature point is determined.

[0191] Here, assuming that φ is a smooth mapping between two smooth manifolds (M, g) and (N, h), then there is formula (1-11):

[0192]

[0193] Converting to the current mesh can obtain formula (1-12):

[0194]

[0195] Wherein, v q is the qth internal point of the re-triangulated triangular mesh model, v t is the adjacent point v q of v qt , k1, k2 are the positions of the two vertices of the edge (q, t) mapped into the projection plane, L q represents the edge connecting v t and v qt , k1, k2 are the positions of the two vertices of the edge (q, t) mapped into the projection plane, L t represents the length of the edge (q, t), represents the area of the triangular patch f(q, t, k).

[0196] Taking the partial derivative of E(v) with respect to v(q) can obtain formula (1-13):

[0197]

[0198] Here, in formula (1-13), v t may refer to the original texture coordinate of the adjacent point of the qth feature point in the re-triangulated triangular patch, and through formula (1-13) and vi new texture coordinates.

[0199] By the steps S1051 to S1055, when the moving distance of the feature point is within the adjustment distance threshold range, it indicates that the moving distance of the feature point is small, and the texture coordinates of the feature point can be kept unchanged, that is, the original texture coordinates of the feature point are determined as the target texture coordinates, so as to reduce the calculation complexity while ensuring the correctness of the calculation result; when the moving distance of the feature point exceeds the adjustment distance threshold, it indicates that the moving distance of the feature point is large, and the target texture coordinates of the feature point need to be determined according to the texture coordinates of the feature points near the target position of the feature point after moving, and in implementation, the texture coordinates with the smallest texture space deformation can be obtained by using the harmonic mapping algorithm to ensure the smooth deformation of large-scale deformation.

[0200] Based on the foregoing embodiments, the embodiments of the present application further provide an image processing method, Figure 5 Another implementation flow diagram of the image processing method provided by the embodiments of the present application is shown in FIG. 5, and the flow includes: Figure 5

[0201] In step S501, the terminal acquires a to-be-processed image and acquires a plurality of feature points of the to-be-processed image.

[0202] Here, the to-be-processed image can be a real person image, can also be a cartoon character image, can also be an animal image, etc., and the to-be-processed image at least includes a face image region. The plurality of feature points acquired in step S501 at least include face feature points of the face image region, can further include edge feature points of the to-be-processed image, and further include feature points on a circle outside the contour of the face image region.

[0203] In step S502, the terminal performs triangular partitioning on the plurality of feature points of the to-be-processed image to obtain index information of a plurality of triangular patches.

[0204] In step S503, the terminal determines a target feature point to be moved and a target position of the target feature point based on the received feature point moving instruction.

[0205] Here, the target feature point to be moved is determined based on the feature point moving instruction, and in implementation, can be determined according to the action position of the moving operation triggering the feature point moving instruction, or can be determined according to the setting information triggering the feature point moving instruction.

[0206] In step S504, the terminal determines a first distance between each target feature point and each other feature point based on the original positions of the other feature points and the original positions of the target feature points.

[0207] ​In step S505, the terminal determines an influence weight of each target feature point on the other feature points based on the respective first distances and a preset influence distance threshold.

[0208] Here, when the first distance is less than or equal to the influence distance threshold, the influence weight of the target feature point on the other feature point is determined according to a ratio of the first distance to the influence distance threshold and a preset influence function; when the first distance is greater than the influence distance threshold, a preset value is determined as the influence weight of the target feature point on the other feature point, and the preset value can be 0.

[0209] In step S506, the terminal determines a candidate target position of the other feature points based on the original position, the target position and the influence weight of each target feature point.

[0210] Here, a target vector of each facial feature point is determined according to the original position and the target position of each target feature point, a candidate moving vector of the other feature points is determined according to the target vector of each target feature point and the corresponding influence weight, and a candidate target position of the other feature points is determined according to the candidate moving vector of the other feature points and the original position.

[0211] In step S507, the terminal determines a second moving distance of the other feature points based on the original position and the candidate target position of the other feature points.

[0212] In step S508, the terminal determines whether the second moving distance of the other feature points is greater than a preset adjustment distance threshold.

[0213] Here, when the second moving distance of the other feature points is greater than the adjustment distance threshold, step S509 is entered; when the second moving distance of the other feature points is less than or equal to the adjustment distance threshold, step S511 is entered.

[0214] In step S509, the terminal determines an attenuation value of the other feature points based on the second moving distance and an attenuation function.

[0215] Here, in the implementation of step S509, an extension distance (i.e., the second distance in other embodiments) of the other feature points when extending to the edge of the image in the moving direction thereof is first determined, then a parameter of the attenuation function is determined based on a ratio of the second moving distance to the second distance, and the attenuation value of the other feature points is determined.

[0216] In step S510, the terminal determines the target position of the other feature points based on the original position, the candidate target position, the adjustment distance threshold and the attenuation value of the other feature points.

[0217] The step S510, when implemented, can be that the other feature points first move an adjusted distance threshold according to the original moving direction, and then outside the adjusted distance threshold, the remaining moving distance can be attenuated according to the attenuation value, so as to determine the target position of the other feature points.

[0218] The step S511, the terminal determines the candidate target position of the other feature points as the target position of the other feature points.

[0219] The step S512, the terminal determines the original texture coordinate of the qth feature point based on the original position of the qth feature point and the size information of the to-be-processed image.

[0220] Here, since the to-be-processed image is a two-dimensional image, the texture coordinate of each pixel point is a real number from 0 to 1, and then the original texture coordinate of the qth feature point can be obtained by dividing the position coordinate of the qth feature point by the width or height of the to-be-processed image.

[0221] The step S513, the terminal determines whether the first moving distance of the qth feature point is greater than the adjusted distance threshold.

[0222] Here, when the first moving distance of the qth feature point is less than or equal to the adjusted distance threshold, the step S514 is entered, and when the first moving distance of the qth feature point is greater than the adjusted distance threshold, the step S515 is entered.

[0223] The step S514, the terminal determines the original texture coordinate of the qth feature point as the target texture coordinate of the qth feature point.

[0224] The step S515, the terminal re-performs the triangulation based on the target positions of the feature points, to obtain index information of a plurality of triangular patches.

[0225] The step S516, the terminal determines the target texture coordinate of the qth feature point based on the index information of the plurality of triangular patches and the harmonic mapping algorithm.

[0226] The step S517, the terminal renders a processed image based on the target positions of the feature points and the target texture coordinates of the feature points.

[0227] The step S518, the terminal sends an image upload request to the server.

[0228] Here, the image upload request carries the processed image, and is used to request to publish the processed image.

[0229] The step S519, the server acquires the processed image carried in the image upload request, audits the processed image, and publishes the processed image after the audit is passed.

[0230] In the image processing method provided in this application embodiment, after the terminal acquires the image to be processed, it acquires facial feature points, image edge feature points, and facial contour feature points, and then performs triangulation. Based on the movement operation of certain feature points, these feature points (i.e., target feature points) are moved. Furthermore, the movement of the target feature points affects other feature points near the target feature points, and when determining the final target position of other feature points, the influence weight of the target feature point on other feature points is determined based on the distance between the target feature point and other feature points. Then, vector synthesis is performed. Finally, when the movement distance is large, the position of other feature points is adjusted based on the attenuation mechanism. This can achieve large-scale smooth deformation. When calculating texture coordinates, the texture coordinates of feature points whose movement distance is within the adjustment distance threshold remain unchanged. For feature points whose movement distance is outside the adjustment distance threshold, triangulation is performed again, and multiple triangular facets can be used to obtain texture coordinates with minimal deformation in texture space by referring to the principle of two-dimensional harmonic mapping, thus avoiding texture disorder.

[0231] The following will describe an exemplary application of the embodiments of this application in a real-world application scenario.

[0232] Figure 6 This is a schematic diagram illustrating another implementation of the image processing method provided in the embodiments of this application, such as... Figure 6 As shown, the process includes:

[0233] Step S601: Obtain the original face image.

[0234] Here, the original face image can be a real face image or a cartoon character's face image; the original face image can be a color image or a grayscale image.

[0235] Step S602: Extract feature points.

[0236] In step S602, facial feature extraction can be performed using facial feature point recognition algorithms such as the dlib algorithm to obtain several facial feature points. The extracted facial feature points cannot cover areas such as the forehead and cheeks of the face. Figure 7 This is a schematic diagram of facial image feature points provided in an embodiment of this application, such as... Figure 7 As shown, a total of 68 feature points were extracted. Feature points 1 to 17 are cheek feature points, feature points 18 to 27 are eyebrow feature points, feature points 28 to 36 are nose feature points, feature points 37 to 48 are eye feature points, and feature points 49 to 68 are mouth feature points.

[0237] Step S603: Interpolate feature points.

[0238] In the process of feature point interpolation, the forehead edge needs to be interpolated, the cheek feature points need to be interpolated, and the image boundary and the points outside the contour also need to be interpolated.

[0239] In the process of forehead point interpolation, the 28th point and the 31st point on the nasal ala centerline can be used to interpolate the middle forehead point M head_center according to formula (2-1), and the forehead vertex M head is interpolated according to formula (2-2).

[0240] M head_center = α1M 28 + α2M 31 (2-1);

[0241] M head = β1M 28 + β2M head_center (2-2);

[0242] Wherein, α1, α2, β1, β2 are interpolation coefficients, which can be pre-set.

[0243] The points on both sides of the forehead can be interpolated by the forehead vertex and the middle forehead point, which are respectively deflected to the left and right by θ i , that is, the middle forehead point M head_center (x head_center ,y head_center ) is counterclockwise rotated by θ i and clockwise rotated by θ i around the forehead vertex M head (x head ,y head ) to obtain the coordinates of the points on both sides of the forehead. Wherein, θ i can be determined by formula (2-3):

[0244]

[0245] In this way, a series of forehead points M i are obtained, and the cheek points H i can be interpolated by the points of the face contour and the feature points of the nasal ala.

[0246] In the face feature point extraction, only the feature points of the face part are extracted. In the embodiment of the present application, in order to perform the triangle division on the entire image for rendering, the feature points of the image edge need to be obtained. In the implementation, the feature points of the image edge can be obtained by equally sampling each pixel point on the image edge. In addition, in order to divide the face contour and the image boundary more finely and to make the deformation more smooth, the feature points of a circle outside the face contour also need to be interpolated. In the implementation, the interpolated feature points of the face contour can be moved equally but in different directions to obtain the feature points of the circle outside the face contour.

[0247] In step S604, the interpolated feature points are divided into triangles to obtain the triangle patch indexes connected to the feature point set.

[0248] The interpolated feature points constitute the feature point set of the face image. In the implementation of step S604, the Delaunay triangle division algorithm can be used to divide the face image based on the feature point set to obtain the triangle patch indexes connected to the feature point set. Each triangle patch index can be the serial number of the three vertices of the triangle patch in the feature point set. Figure 8 The image diagram obtained by dividing the interpolated feature points into triangles is provided in the embodiment of the present application.

[0249] In step S605, the user defines the constraint points.

[0250] In the embodiment of the present application, the constraint points refer to the points dragged by the user, and the free points refer to the other points moved due to the movement of the constraint points.

[0251] In step S606, the image is rendered based on OpenGL.

[0252] In step S607, the deformation position is calculated when the user drags the constraint points.

[0253] Here, in step S607, the deformation positions of the constraint points and the free points are calculated. The deformation positions and the texture coordinates of the constraint points and the free points are recalculated as soon as the user drags. In the implementation of step S607, the deformation position calculation is based on the vector synthesis of the attenuation mechanism.

[0254] Suppose that the constraint point N1 moves from the original position (x1 src ,y1 src ) to the deformation position (x1 dst ,y1 dst ), and the constraint point N2 moves from the original position (x2 src ,y2 src ) to the deformation position (x2 dsty2 dst , constraint point N n n moves from (xn src ,yn src ) to deformed position (xn dst ,yn dst ), the movement of the constraint point is regarded as the action of force on the free point, the closer to a certain constraint point, the greater the influence of this constraint point on the free point, as shown in Figure 9 , the free point Pi is closest to the constraint point Nn, so the influence of Nn on the point is greater than N1, N2, N3, so the moving direction of Pi is closest to Nn.

[0255] Assuming that the free points P1(s1 src ,t1 src ), P2(s2 src ,t2 src ), … P m (sm src ,tm src ) are deformed after the position of the constraint point P'1(s1 dst ,t1 dst ), P'2(s2 dst ,t2 dst ), … P' m (sm dst ,tm dst ), then the vector of Pj moving from the starting position Pj to the target position Pj' can be calculated by formula (2-4):

[0256]

[0257] wherein is the movement vector of the constraint point Ni, w i (r) is the Gaussian weight function, that is, the influence weight in other embodiments, w i (r) can be determined by formula (2-5):

[0258]

[0259] wherein d is the distance from the constraint point to the free point, δ is the influence domain radius, formula (2-5) shows that the influence decreases to 0 after the free point and the constraint point exceed a certain range, that is, each constraint point only has an influence on the free points within a certain range.

[0260] ​Due to considering large-scale drag, the influence applied to the free point in the influence domain of the free point should be increased with the increase of the drag distance, and if the influence domain is exceeded, it should be slowly attenuated with the increase of the drag distance, so that the deformation can be smoothed and the problem of large-scale deformation can be adaptively solved, and therefore, in the embodiment of the present application, the attenuation function shown in formula (2-6) is designed

[0261]

[0262] In the embodiment of the present application, a deformation domain of each free point can be set, which is a circular deformation domain, assuming that the radius is R, and σ is an attenuation degree parameter, which can be a preset value.

[0263] The boundary of the image is taken as the limit of deformation, and the action direction of the constraint point to the free point P i is obtained through the synthesis of formula (2-4), and then the moving direction of the free point and the intersection point of the image boundary are determined, which are set as G i (x border ,y border ), and s in formula (2-6) can be determined through formula (2-7):

[0264]

[0265] wherein, l is the distance of the free point from the original position, that is When the influence domain is exceeded, g(s) is increased with the increase of l, so that the final deformation vector of the free point can be determined according to formula (2-8):

[0266]

[0267] G1(v) is the final deformation vector of the free point, and through the deformation vector and the original position of the free point, the final deformation position of the free point can be determined.

[0268] In actual implementation, the vector field synthesis deformation position calculation algorithm in this step can be replaced by ml s deformation, thin plate spline, etc., but compared with the deformation algorithm based on vector field synthesis, the deformation algorithm based on vector field synthesis is faster and can smooth the deformation.

[0269] In step S608, texture coordinate calculation is performed.

[0270] In step S609, image rendering is performed again based on OpenGL, and a deformed image is obtained.

[0271] Since the user drags, the dragged constraint point can move far away from the original position, which is assumed as O, and other free points following the motion of the constraint point move within their own deformation domain, so the texture coordinates of the dragged constraint point and the free points are certainly different. When the dragged constraint point exceeds the deformation domain, the texture coordinates should be determined by the texture coordinates of the constraint points near O, and the texture coordinates of the free points within the deformation domain remain unchanged, so that the stretching of the texture generates the deformation. Here, for the dragged control point, the grid topology can be used to obtain the texture coordinates with the minimum texture space deformation by referring to the principle of two-dimensional harmonic mapping.

[0272] Suppose the position of the constraint point before deformation is s(x pos ,y pos ), and the position of the control point after dragging deformation is s'(x pos ,y' pos ).

[0273] Since the face image is a two-dimensional image, the texture coordinates of all points T(x, y) can be calculated by formula (2-9):

[0274]

[0275] where width is the width of the original face image, and height is the height of the original face image.

[0276] In the embodiment of the present application, the texture coordinates of the deformed feature points can be achieved by formula (2-10):

[0277]

[0278] where R is the deformation domain radius, and distance(s', s) is the moving distance before and after deformation. That is, when the moving distance of a feature point before and after deformation is less than R, the texture coordinates after deformation are different from the texture coordinates before deformation, and when the moving distance exceeds the deformation domain, the original triangular facet index is discarded, and a new triangular facet index is obtained by using Delaunay triangulation, because the dragged point has new adjacent points, so that the new texture coordinates of the point are calculated by using the new adjacent points and harmonic texture mapping. The effect is as shown in Figure 10 , where 1001 is the face image before deformation, and when a point on the cheek is dragged at a large scale, if the texture is only kept unchanged, the dragged point will have texture errors, as shown in 1002 in Figure 10 , and adaptive texture coordinate calculation can achieve better results, as shown in 1003 in Figure 10 .

[0279] In order to better understand the embodiments of the present application, the harmonic texture mapping is described here.

[0280] Assume that φ is a smooth mapping between two smooth manifolds (M,g) and (N,h), then there is formula (2-11):

[0281]

[0282] Converting to the current mesh can obtain formula (2-12):

[0283]

[0284] wherein, v i (i = 1, 2, …, n) is an internal point of the triangular mesh model, v ij represents the edge connecting v i and v j , k1, k2 are the positions of the two vertices of the edge (i, j) mapped into the projection plane, L ij represents the length of the edge (i, j), represents the area of the triangular patch f(i, j, k).

[0285] Taking the partial derivative of E(v) with respect to v(i) can obtain formula (2-13):

[0286]

[0287] The new texture coordinates of v i can be determined through formula (2-13).

[0288] By the image processing method provided in the embodiments of the present application, various natural smooth large-scale deformation effects can be provided for face special effect products, and the face deformation provides a key basic technology for subsequent face animation, face fusion, or face changing applications.

[0289] The image processing apparatus 354 provided in the embodiments of the present application will be described below as an example of a software module. In some embodiments, as shown in FIG. 3, the software module stored in the image processing apparatus 354 of the memory 350 can be an image processing apparatus in the terminal 300, which includes: Figure 2

[0290] The first acquisition module 3541 is configured to acquire a to-be-processed image, and perform feature extraction on the to-be-processed image to obtain a plurality of image feature points.

[0291] The interpolation processing module 3542 is configured to perform interpolation processing on the plurality of image feature points to obtain a plurality of interpolated feature points.

[0292] ​The first determining module 3543 is configured to determine a target feature point to be moved and a target position of the target feature point based on the received feature point movement instruction.

[0293] The second determining module 3544 is configured to determine target positions of other feature points based on the original position and the target position of the target feature point, the original positions of the other feature points, and a preset attenuation function, the other feature points being each feature point except the target feature point.

[0294] The third determining module 3545 is configured to determine a first movement distance of each feature point, and determine a target texture coordinate of each feature point based on the first movement distance, the original position of each feature point, and a harmonic texture mapping algorithm.

[0295] The rendering module 3546 is configured to render a processed image based on the target positions of the feature points and the target texture coordinates of the feature points.

[0296] In some embodiments, the second determining module is further configured to:

[0297] determine a first distance between each target feature point and each other feature point based on the original position of each target feature point and the original position of each other feature point;

[0298] determine an influence weight of each target feature point on each other feature point based on the first distance and a preset influence distance threshold;

[0299] determine a candidate target position of each other feature point based on the original position and the target position of each target feature point, and the influence weight;

[0300] determine the target position of each other feature point based on the original position, the candidate target position of each other feature point, and a preset attenuation function.

[0301] In some embodiments, the second determining module is further configured to:

[0302] when the ith first distance is less than or equal to the influence distance threshold, determine a ratio of the ith first distance to the influence distance threshold as a first parameter of an influence function; i = 1, 2, … N, N being a total number of target feature points;

[0303] determine an influence weight of the ith target feature point on the other feature points based on a second parameter of the influence function and the first parameter;

[0304] when the ith first distance is greater than the influence distance threshold, determine a preset value as the influence weight of the ith target feature point on the other feature points.

[0305] In some embodiments, the second determining module is further configured to:

[0306] determine a target vector for each target feature point based on the original position of the target feature point and the corresponding target position;

[0307] determine a candidate moving vector for the other feature point based on the influence weight of the target feature point on the other feature point and the target vector;

[0308] determine a candidate target position of the other feature point based on the candidate moving vector and the original position of the other feature point.

[0309] In some embodiments, the second determining module is further configured to:

[0310] determine a second moving distance of the other feature point based on the original position and the candidate target position of the other feature point;

[0311] determine an attenuation value of the other feature point based on the second moving distance and an attenuation function when the second moving distance is greater than a preset adjustment distance threshold;

[0312] determine a target position of the other feature point based on the original position, the candidate target position, the adjustment distance threshold and the attenuation value of the other feature point.

[0313] In some embodiments, the second determining module is further configured to:

[0314] determine the candidate target position as the target position of the other feature point when the second moving distance is less than or equal to the adjustment distance threshold.

[0315] In some embodiments, the second determining module is further configured to:

[0316] determine a unit vector of the other feature point in a moving direction based on the original position and the candidate target position of the other feature point;

[0317] determine a third parameter outside the adjustment distance threshold based on the second moving distance, the adjustment distance threshold and the attenuation value, and determine the adjustment distance threshold as a fourth parameter;

[0318] determine a target moving vector of the other feature point based on the third parameter, the fourth parameter and the unit vector;

[0319] determine a target position of the other feature point based on the target moving vector and the original position of the other feature point.

[0320] In some embodiments, the interpolation processing module is further configured to:

[0321] interpolating the extracted image feature points to obtain interpolated image feature points;

[0322] determining contour feature points of a target image region based on the interpolated image feature points, and obtaining image edge feature points of the image to be processed;

[0323] determining the interpolated image feature points, the contour feature points and the image edge feature points as a plurality of interpolated feature points;

[0324] In some embodiments, the apparatus further comprises:

[0325] triangulation module, configured to triangulate a plurality of feature points of the image to be processed to obtain index information of a plurality of triangular patches.

[0326] In some embodiments, the third determining module is further configured to:

[0327] determine original texture coordinates of the qth feature point based on the original position of the qth feature point and size information of the image to be processed;

[0328] when the first movement distance of the qth feature point is less than or equal to the adjustment distance threshold, determine the original texture coordinates of the qth feature point as target texture coordinates of the qth feature point.

[0329] In some embodiments, the third determining module is further configured to:

[0330] when the first movement distance of the qth feature point is greater than the adjustment distance threshold, re-triangulate based on the target positions of the feature points to obtain index information of a plurality of triangular patches;

[0331] determine target texture coordinates of the qth feature point based on the index information of the plurality of triangular patches and a harmonic mapping algorithm.

[0332] It should be noted that the description of the device embodiments of the present application is similar to that of the above-mentioned method embodiments, and has similar beneficial effects as the method embodiments, and therefore will not be described here. For technical details not disclosed in the device embodiments, please refer to the description of the method embodiments for understanding.

[0333] The embodiments of the present application provide a storage medium storing executable instructions, wherein the executable instructions are stored, and when the executable instructions are executed by a processor, the processor will execute the method provided by the embodiments of the present application, for example, the method shown in Figure 4 .

[0334] In some embodiments, the storage medium can be a computer-readable storage medium, such as a ferroelectric memory (FRAM), a read only memory (ROM), a programmable read only memory (PROM), an erasable programmable read only memory (EPROM), an electrically erasable programmable read only memory (EEPROM), a flash memory, a magnetic surface storage, an optical disc, or a compact disc read only memory (CD-ROM), and the like. It can also be various devices including one or any combination of the above memories.

[0335] In some embodiments, the executable instructions can be in the form of programs, software, modules, scripts, or code, written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0336] By way of example, the executable instructions can or can not correspond to a file in a file system, can be stored in a part of a file that holds other programs or data, can be stored as one or more scripts or set of instructions in a markup language such as Hyper Text Markup Language (HTML), may

[0337] The above merely provides an example of the embodiments of the present application, and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, and improvement within the spirit and scope of the present application shall be included in the protection scope of the present application.

Claims

1. An image processing method, characterized by, The method comprises: acquiring a to-be-processed image, and performing feature extraction on the to-be-processed image to obtain a plurality of image feature points; performing interpolation processing on the plurality of image feature points to obtain a plurality of interpolated feature points; based on the received feature point movement instruction, determining a target feature point to be moved and a target position of the target feature point; based on the original positions of the other feature points and the original positions of the target feature points, determining a first distance between each of the other feature points and each of the target feature points, wherein the other feature points are each of the feature points other than the target feature points; based on the first distance and a preset influence distance threshold, determining an influence weight of each of the target feature points on the other feature points; based on the original positions of each of the target feature points, the target position and the influence weight, determining a candidate target position of the other feature points; based on the original position of the other feature points, the candidate target position and a preset attenuation function, determining a target position of the other feature points; determining a first movement distance of each of the feature points, and based on the first movement distance, the original position of each of the feature points and a harmonic texture mapping algorithm, determining a target texture coordinate of each of the feature points; based on the target position of each of the feature points and the target texture coordinate of each of the feature points, rendering a processed image.

2. The method of claim 1, wherein, The method further comprises: when the first distance is less than or equal to the influence distance threshold, determining a ratio of the first distance to the influence distance threshold as a first parameter of an influence function; based on a second parameter of the influence function and the first parameter, determining the influence weight of the target feature point on the other feature points; when the first distance is greater than the influence distance threshold, determining a preset value as the influence weight of the target feature point on the other feature points.

3. The method of claim 2, wherein, The method further comprises: based on the original position of each of the target feature points and the corresponding target position, determining a target vector; based on the influence weight of each of the target feature points on the other feature points and the target vector, determining a candidate movement vector corresponding to the other feature points; based on the candidate movement vector and the original position of the other feature points, determining a candidate target position of the other feature points.

4. The method of claim 1, wherein, The method further comprises: based on the original position of the other feature points and the candidate target position, determining a second movement distance of the other feature points; when the second movement distance is greater than a preset adjustment distance threshold, based on the second movement distance and the attenuation function, determining an attenuation value of the other feature points; determine the target position of the other feature point based on the original position of the other feature point, the candidate target position, the adjustment distance threshold and the attenuation value.

5. The method of claim 4, wherein, The method further comprises: when the second movement distance is less than or equal to the adjustment distance threshold, determining the candidate target position as the target position of the other feature point.

6. The method as claimed in claim 4, wherein, The method further comprises: when the second movement distance is less than or equal to the adjustment distance threshold, determining the candidate target position as the target position of the other feature point. The method further comprises: determining a unit vector of the other feature point in a movement direction based on the original position and the candidate target position of the other feature point; determining a third parameter outside the adjustment distance threshold based on the second movement distance, the adjustment distance threshold and the attenuation value, and determining the adjustment distance threshold as a fourth parameter; 7. The method as claimed in claim 1, wherein, determining a target movement vector of the other feature point based on the third parameter, the fourth parameter and the unit vector; determining the target position of the other feature point based on the target movement vector and the original position of the other feature point. The method further comprises: performing interpolation processing on the extracted image feature points to obtain interpolated image feature points; determining contour feature points of a target image region based on the interpolated image feature points, and obtaining image edge feature points of the image to be processed; determining the interpolated image feature points, the contour feature points and the image edge feature points as the plurality of interpolated feature points; 8. The method of claim 7, wherein, The method further comprises: triangulating the image to be processed based on the plurality of interpolated feature points to obtain index information of a plurality of triangular patches. The method further comprises:

9. The method of claim 8, wherein, determining the original texture coordinates of the qth feature point based on the original position of the qth feature point and the size information of the image to be processed; when the first movement distance of the qth feature point is less than or equal to the adjustment distance threshold, determining the original texture coordinates of the qth feature point as the target texture coordinates of the qth feature point. The method further comprises:

10. An image processing apparatus characterized by comprising: when the first movement distance of the qth feature point is greater than the adjustment distance threshold, re-performing triangulation based on the target positions of the feature points to obtain index information of a plurality of triangular patches; determining the target texture coordinates of the qth feature point based on the index information of the plurality of triangular patches and the harmonic texture mapping algorithm. The method further comprises: a first obtaining module configured to obtain an image to be processed, and perform feature extraction on the image to be processed to obtain a plurality of image feature points; an interpolation processing module configured to perform interpolation processing on the plurality of image feature points to obtain a plurality of interpolated feature points; a first determining module configured to determine a target feature point to be moved and a target position of the target feature point based on a received feature point movement instruction; determine a first distance between each of the target feature points and the other feature points based on the original positions of the other feature points and the original positions of the target feature points; determine an influence weight of each of the target feature points on the other feature points based on the first distance and a preset influence distance threshold; determine a candidate target position of the other feature points based on the original positions of the target feature points, the target position and the influence weight; determine a target position of the other feature points based on the original positions of the other feature points, the candidate target position and a preset attenuation function; determine a first movement distance of each of the feature points, and determine a target texture coordinate of each of the feature points based on the first movement distance, the original position and a harmonic texture mapping algorithm of each of the feature points; render a processed image based on the target position of each of the feature points and the target texture coordinate of each of the feature points.

11. The apparatus of claim 10, wherein, The second determining module is further configured to: when the first distance is less than or equal to the influence distance threshold, determine a ratio of the first distance to the influence distance threshold as a first parameter of an influence function; determine the influence weight of each of the target feature points on the other feature points based on a second parameter of the influence function and the first parameter; when the first distance is greater than the influence distance threshold, determine a preset value as the influence weight of each of the target feature points on the other feature points.

12. The apparatus of claim 10, wherein, The second determining module is further configured to: determine a target vector based on the original position and the target position of each of the target feature points; determine a candidate movement vector of the other feature points based on the influence weight of each of the target feature points on the other feature points and the target vector; determine the candidate target position of the other feature points based on the candidate movement vector and the original position of the other feature points.

13. The apparatus of claim 10, wherein, The second determining module is further configured to: determine a second movement distance of the other feature points based on the original position and the candidate target position of the other feature points; when the second movement distance is greater than a preset adjustment distance threshold, determine an attenuation value of the other feature points based on the second movement distance and the attenuation function; determine the target position of the other feature points based on the original position, the candidate target position, the adjustment distance threshold and the attenuation value of the other feature points.

14. The apparatus of claim 13, wherein, The second determining module is further configured to: when the second movement distance is less than or equal to the adjustment distance threshold, determine the candidate target position as the target position of the other feature points.

15. The apparatus of claim 13, wherein, The second determining module is further configured to: determine a unit vector of the other feature points in a movement direction based on the original position and the candidate target position of the other feature points; determine a third parameter outside the adjustment distance threshold based on the second movement distance, the adjustment distance threshold and the attenuation value, and determine the adjustment distance threshold as a fourth parameter. determine a target movement vector of the other feature point based on the third parameter, the fourth parameter and the unit vector; determine a target position of the other feature point based on the target movement vector and the original position of the other feature point.

16. The apparatus of claim 10, wherein, The interpolation processing module is further configured to: perform interpolation processing on the extracted image feature points to obtain interpolated image feature points; determine contour feature points of a target image region based on the interpolated image feature points, and acquire image edge feature points of the image to be processed; determine the interpolated image feature points, the contour feature points and the image edge feature points as a plurality of interpolated feature points; perform triangulation on the image to be processed based on the plurality of interpolated feature points to obtain index information of a plurality of triangular patches.

17. The apparatus of claim 16, wherein, The third determining module is further configured to: determine an original texture coordinate of the qth feature point based on the original position of the qth feature point and the size information of the image to be processed; when the first movement distance of the qth feature point is less than or equal to an adjustment distance threshold, determine the original texture coordinate of the qth feature point as a target texture coordinate of the qth feature point.

18. The apparatus of claim 17, wherein, The third determining module is further configured to: when the first movement distance of the qth feature point is greater than the adjustment distance threshold, perform triangulation again based on the target positions of the feature points to obtain index information of a plurality of triangular patches; determine the target texture coordinate of the qth feature point based on the index information of the plurality of triangular patches and a harmonic texture mapping algorithm.

19. An image processing apparatus characterized by comprising: comprise: a memory configured to store executable instructions; a processor configured to execute the executable instructions stored in the memory to implement the method in any one of claims 1 to 9.

20. A computer-readable storage medium, characterized in that, executable instructions stored in the memory, which are configured to cause the processor to implement the method in any one of claims 1 to 9 when executed.

Images