Sticker generation method, device, medium and electronic device

Abstract

The present disclosure provides a sticker generation method, including: obtaining a background image, wherein the background image includes a target object, displaying a sticker display area and a sticker anchor point in the background image, receiving an import instruction for the sticker, importing sticker resources according to the import instruction and displaying them in the sticker display area, dynamically selecting a tracking area according to the sticker anchor point position, wherein the tracking area is an image area in the target object, and generating the sticker according to the display area, the tracking area, and the sticker resources. Through the sticker generation technology disclosed in the present disclosure, when a sticker moves on a target object, it is possible to select an image area in the target object in real time, more accurately determine the relative positional relationship between the sticker and the target object, more accurately track the target object, accurately set the sticker position, and dynamically adjust the sticker position according to the target object detection result, thereby achieving a real-time dynamic sticker effect with quick response.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method, device, medium and electronic device for generating stickers. Background Art

[0002] With the development of mobile Internet technology, especially the development of images, image processing and recognition are becoming more and more intelligent, and various beauty cameras and special effects software are becoming more and more popular. In order to diversify the image effects, people will create some special effects on the images to increase the interest of the images. Now the more popular sticker special effects are very popular, such as setting cat ears or cat whiskers on people's faces.

[0003] Currently, when making 2D sticker effects, two key points need to be set on the face so that the 2D sticker can track these two key points to track the movement of the face. However, sometimes the two points set manually are not accurate and cannot adapt to various situations. For example, if the 2D sticker is a hat and the two key points to be tracked are set to the eyes, when the person lowers his head, the key points of the eyes cannot be detected, which may lead to the situation that the tracking cannot be achieved. Summary of the invention

[0004] In order to track faces more accurately and precisely set the relative position of stickers and faces, the purpose of the present invention is to provide a sticker generation method, device, medium and electronic device that can solve at least one of the technical problems mentioned above.

[0005] The present disclosure embodiment specifically provides a sticker generation method, which includes:

[0006] Acquire a background image, wherein the background image includes a target object;

[0007] Displaying a display area of ​​the sticker and an anchor point of the sticker in the background image;

[0008] Receive an import instruction for the sticker;

[0009] Import the resources of the sticker according to the import instruction and display them in the display area of ​​the sticker;

[0010] Dynamically selecting a tracking area according to the anchor point position of the sticker, wherein the tracking area is an image area in the target object; and generating the sticker according to the display area, the tracking area and the resources of the sticker.

[0011] According to a specific embodiment of the present invention, on the other hand, the present invention provides a sticker generating device, comprising:

[0012] A background acquisition unit, used to acquire a background image, wherein the background image includes a target object;

[0013] A sticker display unit, used to display a sticker display area and a sticker anchor point in the background image;

[0014] An import receiving unit, used for receiving an import instruction of the sticker;

[0015] A sticker importing unit, used to import sticker resources according to the import instruction and display them in the sticker display area;

[0016] An area selection unit is used to dynamically select a tracking area according to the anchor point position of the sticker, wherein the tracking area is an image area in the target object; and a sticker generation unit is used to generate the sticker according to the display area, the tracking area and the resources of the sticker.

[0017] According to a specific embodiment of the present invention, on the other hand, the present invention provides an electronic device, comprising: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the sticker generation method as described in any one of the above items. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the following is a brief introduction to the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0019] Figure 1 A flowchart of a sticker generation method according to an embodiment of the present invention is shown;

[0020] Figure 2 A schematic diagram showing the meaning of a canvas according to an embodiment of the present invention is shown;

[0021] Figure 3 A schematic diagram of a 2D canvas element according to an embodiment of the present invention is shown;

[0022] Figure 4 A segmentation schematic diagram of a triangulation algorithm (Delaunay) according to an embodiment of the present invention is shown;

[0023] Figure 5a-5b A diagram showing the position relationship of a triangular area formed by anchor points and key points according to an embodiment of the present invention;

[0024] Figure 6 A schematic diagram showing a situation where an anchor point is outside the range of a target object according to an embodiment of the present invention;

[0025] Figure 7 A structural diagram of a sticker generating device according to an embodiment of the present invention is shown.

[0026] Figure 8 A schematic diagram of a connection structure of an electronic device according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0027] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0028] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0029] The term "including" and its variations used in this document are open inclusions, that is, "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". Relevant definitions of other terms will be given in the description below. It should be noted that the concepts of "first", "second", etc. mentioned in this disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0030] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0031] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0032] It should also be noted that the term "includes", "comprising" or any other variation thereof is intended to cover non-exclusive inclusion, so that a commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprising a ..." do not exclude the existence of other identical elements in the commodity or device including the elements.

[0033] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein may be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on the present disclosure, it should be understood by those skilled in the art that an aspect described herein may be implemented independently of any other aspect, and two or more of these aspects may be combined in various ways. For example, any number of aspects described herein may be used to implement the device and / or practice the method. In addition, other structures and / or functionalities other than one or more of the aspects described herein may be used to implement this device and / or practice this method.

[0034] It should also be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present disclosure. The drawings only show components related to the present disclosure rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0035] Additionally, in the following description, specific details are provided to facilitate a thorough understanding of the examples. However, it will be understood by those skilled in the art that the aspects described may be practiced without these specific details.

[0036] The optional embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0037] Example 1

[0038] Attached Figure 1 This is a flowchart of an embodiment of a sticker generation method provided by the present disclosure. The sticker generation method provided by this embodiment is used to set sticker special effects on a two-dimensional image avatar. The sticker generation method requires an image processing device to execute. The image processing device can be implemented as software, or as a combination of software and hardware. The expression image effect generation device can be integrated in a device in the image processing system, such as an image processing server or an image processing terminal device. Figure 1 As shown, the method comprises the following steps:

[0039] Step S101, obtaining a background image, wherein the background image includes a target object. The image in this embodiment may be a static picture, a dynamic picture, or a video stream or a real-time video shot, wherein the image includes a target object for which a sticker special effect is required. Usually, the target object may be the face or head of a person or an animal. The image processing device receives an image acquisition instruction from the client, and obtains a background image to be processed by opening and calling an image or camera device. The background image may be a stored picture or video, or a picture or video shot in real time, as a background image for setting stickers.

[0040] Step S102, displaying the display area of ​​the sticker and the anchor point of the sticker in the background image. In the embodiment of the present invention, the background image selects a face, and the sticker effect is actually to stick some pictures on the face, and these pictures change with the position of the face. The position of the sticker is dynamically adjusted according to the result of face detection, so as to achieve a simple real-time dynamic sticker effect. Among them, the position of the sticker is determined by the anchor point of the sticker. The position of the anchor point on the canvas determines the position of the sticker in the canvas, thereby determining the corresponding position of the sticker and the background image. In this embodiment, the center point of the rectangular area where the sticker is located or one end point of the rectangular area is defined as the anchor point of the sticker, and the anchor point coordinates O (x0, y0) are set in the image. The set anchor point can be dragged, and the position can be set in advance according to the type of sticker. This embodiment introduces how to set the relative position of an anchor point coordinate and the target object, and update it in real time. If multiple stickers are set, multiple anchor points need to be set.

[0041] Background real-time update step S103, receiving the sticker import instruction. Here, we first introduce two keywords: canvas, 2D panel, Figure 2 and Figure 3 The element definition diagrams of the canvas and 2D stickers are shown respectively, where the canvas is an editable image area and image element, such as Figure 2 As shown, it includes a background image, key points, tracking areas and stickers, etc.; the 2D panel includes a canvas, a ruler, a reference line, a lock, etc. The position of the sticker on the canvas is set by the user, and the sticker can be moved freely within the canvas. The sticker can be dragged and scaled arbitrarily within the canvas. According to the anchor point of the sticker obtained in step 103, the sticker setting instruction sent by the client is received, and the initial position of the anchor point of the sticker is set. The initial position is set by the user, that is, the initial coordinates of the anchor point O (x0, y0). The set anchor point can be dragged, and the position can be set in advance according to the type of sticker.

[0042] Step S104, according to the import instruction, the resources of the stickers are imported and displayed in the display area of ​​the stickers. Stickers can be varied, such as cat ears, cat whiskers, rabbit ears, horns, hats, decorations, graffiti, etc. The types of stickers include single pictures and sequence frame animations. The display mode of the stickers is selected from one of the display modes of forward display, reverse display, angle display, horizontal mirroring and vertical mirroring. Single selection or multiple selection is possible. Multiple stickers can be set in a symmetrical manner by horizontal or vertical mirroring, or can be rotated arbitrarily at a certain angle, and stickers can be set at a certain angle. If the background image is a video, the time sequence can be set in a chronological order. Of course, this setting has certain liberalization and personalization. Stickers are imported into the background image, and the stickers are displayed in the target object of the background image according to the setting instructions. For example, cat ear stickers are displayed in a mirror-symmetrical manner on both sides of the mouth. The cat ear stickers correspond to two rectangular sticker display areas, and the sticker areas are within the range of the target object. In addition, cat ear stickers are displayed in mirror symmetry on both sides of the top of the head, and the cat ear stickers correspond to two rectangular sticker display areas, which are outside the range of the target object.

[0043] After the sticker is displayed in the background image, the position adjustment and size scaling instructions of the sticker are received. The position and size of the sticker are set according to the needs of the user. For example, the sticker can be set on the top of the head, on the mouth, or in other positions, and the size can also be adjusted. The position and size of the sticker can be achieved by dragging. The sticker can be dragged at any point within the canvas range, and the drag operation can be performed in the 2D panel. The dragging of the sticker will adsorb the reference line, including boundary adsorption and central axis adsorption. The dragging of the sticker includes two modes: arbitrary dragging and limited dragging, where limited dragging includes dragging in the X, Y, XY and YX directions. The setting operation of the sticker is implemented on the 2D panel, where the 2D panel can be arbitrarily scaled within the specified range, and the pixel value of the internal elements remains unchanged. The ruler includes two types, horizontal and vertical, which are located on the left and top of the 2D panel respectively, and the position of the 0 point is always consistent with the position of the upper left corner of the canvas. The 2D panel also includes a lock, which is located at the leftmost position of the horizontal ruler. After locking, the reference line cannot be dragged, and the reference line cannot be dragged from the ruler. The reference lines include two types: horizontal and vertical. They appear when dragged from the ruler and disappear when dragged into the ruler. They change their width as the 2D panel is scaled, and their position relative to the upper left corner of the canvas is displayed in real time when dragged.

[0044] Step S105, dynamically select a tracking area according to the anchor point position of the sticker, wherein the tracking area is an image area in the target object. The tracking area is used to display the area corresponding to the sticker in the background image. In order to obtain the tracking area, the face in the background image needs to be segmented first, and the segmentation is performed by identifying key points on the face, wherein the key points are at least three, usually dozens to hundreds.

[0045] In this embodiment, the face of a person is set as the target object for analysis, wherein the facial key point information of the face can reflect the position and direction of the face, and the key point information includes the internal feature key point information and the edge feature key point information. The facial key point detection algorithm in this embodiment integrates the structural information described by the edge line of the face into the key point detection, which greatly improves the detection accuracy of the algorithm in extreme cases such as large side face, exaggerated expression, occlusion, blur, etc. This embodiment obtains the coordinates of 106 key points of the face through the facial key point algorithm provided by the lab. The algorithm obtains the facial key point detection data set Wider Facial Landmarks in-the-wild (WFLW) by capturing the common image feature information of the face, which contains facial image data with key points and facial attributes annotated, including posture, expression, lighting, makeup, occlusion and blur transformation, aiming to help the academic community more targetedly evaluate the robustness of the key point algorithm under various conditions. In principle, the number of key points should not be less than 3. However, if there are too few key points, the feature representation of the target object in the image will be too small, and the features of the target object cannot be accurately reflected, and the position of the target object cannot be accurately located. The more key points there are, the more accurate the feature representation of the target object will be, but the amount of calculation will also be greater. Usually, dozens to hundreds of key points can be selected. For example, 106 can be selected, or 22, 98, 100, and so on.

[0046] Performing regional segmentation on a human face (target object) in a background image includes: using a triangulation algorithm of a key point set to segment the human face (target object) into N triangular regions, wherein N is a natural number.

[0047] Figure 4 A segmentation schematic diagram of the triangulation algorithm (Delaunay) is shown. As described in the figure, discrete key points are connected into Delaunay triangles according to the triangulation algorithm (Delaunay), the target object is divided into multiple triangular areas, and the target object is divided into N triangular areas using the point set triangulation algorithm, where N is a natural number.

[0048] The definition of the Delaunay triangle is:

[0049] Triangulation: Assume that V is a finite set of points in a two-dimensional real number field, edge e is a closed line segment consisting of points in the point set as endpoints, and E is the set of e. Then a triangulation T = (V, E) of the point set V is a planar graph G that satisfies the following conditions:

[0050] 1. Except for the endpoints, the edges in a planar graph do not contain any points in the point set.

[0051] 2. There are no intersecting edges.

[0052] 3. All faces in the plane graph are triangular faces, and the union of all triangular faces is the convex hull of the scattered point set V.

[0053] The most commonly used triangulation in practice is the Delaunay triangulation, which is a special triangulation. Let's start with the Delaunay edge:

[0054] Delaunay edge: Suppose there is an edge e in E (with two endpoints a and b). If e satisfies the following conditions, it is called a Delaunay edge: there exists a circle passing through points a and b, and the circle does not contain any other points in the point set V (note that it is inside the circle, and at most three points on the circle share the same circle). This property is also called the empty circle property.

[0055] Delaunay triangulation: If a triangulation T of a point set V contains only Delaunay edges, then the triangulation is called a Delaunay triangulation.

[0056] Assume T is any triangulation of V, then T is a Delaunay triangulation of V if and only if the interior of the circumcircle of every triangle in T does not contain any point in V.

[0057] There are algorithms for Delaunay triangles, such as edge flanging algorithm, point-by-point insertion algorithm, split-and-merge algorithm, Bowyer-Watson algorithm, etc. This embodiment adopts a point-by-point insertion algorithm, in which Delaunay triangles are mainly used. It can be understood that the circumscribed circle of each triangle cannot contain any other point in the point set.

[0058] The optimized pseudo code is:

[0059]

[0060]

[0061] Thus, a multi-triangle region union composed of multiple key points can be obtained. In this embodiment, the face is divided into 159 triangles (not shown) by Delaunay by using the 106 key points of the face.

[0062] The tracking area is a triangular area formed by three key points in the face (target object). When the anchor point of the sticker is inside the area of ​​the face (target object), the tracking area is a triangular area containing the anchor point; when the anchor point of the sticker is outside the area of ​​the face (target object), the tracking area is the three key point coordinates within the face (target object) that are relatively fixed and best reflect the characteristics of the target object, that is, the two eyes and the end of the nose are selected as the three key points by default. The tracking area is updated in real time according to the position of the anchor point of the sticker in the canvas. The display of the tracking area includes two display modes: display point prompts and no display point prompts.

[0063] Figure 5a-5b The position relationship diagram of the triangle area formed by the anchor point and the key point is shown, such as Figure 5a-5b As shown, according to the coordinates of the sticker anchor point, determine in which triangle the anchor point is located, specifically:

[0064] Select three key point coordinates from the key point coordinates; calculate the area of ​​the large triangle formed by the three key point coordinates; according to the anchor point coordinates, use Heron's formula to calculate the areas of three small triangles formed by the anchor point and every two of the three key point coordinates; if the sum of the areas of the three small triangles is equal to the area of ​​the large triangle, the anchor point is located inside the large triangle; if the sum of the areas of the three small triangles is greater than the area of ​​the large triangle, the anchor point is located outside the large triangle.

[0065] Heron's formula is used to calculate the area of ​​a triangle. For example, if the three key points form a triangle called A(x1, y1), B(x2, y2), and C(x3, y3), and the side lengths of the corresponding triangle are a, b, and c, then the area of ​​the triangle can be obtained according to Heron's formula:

[0066] Calculating the area of ​​a triangle

[0067] Heron's formula

[0068] S ABC =sqrt(p*(pa)*(pb)*(pc)), S ABC is the area of ​​triangle ABC

[0069] in,

[0070] a=sqrt((x1-x2)*(x1-x2)+(y1-y2)*(y1-y2));

[0071] b=sqrt((x1-x3)*(x1-x3)+(y1-y3)*(y1-y3));

[0072] c=sqrt((x3-x2)*(x3-x2)+(y3-y2)*(y3-y2));

[0073] p=(a+b+c) / 2, p is the semi-perimeter of the triangle.

[0074] Similarly, the area of ​​the triangle between the anchor point and every two points of the three key points can be calculated to obtain the areas of the three triangles S1, S2 and S3. If S1+S2+S3=S ABC ,refer to Figure 5a , then the anchor point O is inside triangle ABC if S1+S2+S3>S ABC ,refer to Figure 5b , then the anchor point O is outside triangle ABC.

[0075] The tracking area is determined based on the positional relationship between the anchor point O and the triangle ABC calculated by the above method. When the anchor point is within the target range, the tracking area is determined to be the triangular area containing the anchor point, that is, the anchor point O is located inside the triangle ABC, and is updated in real time as the anchor point position changes.

[0076] Figure 6 The diagram shows the situation where the anchor point is outside the target object. When the sticker is outside the target object (e.g., a face), refer to the attached Figure 4 When the anchor point is outside the target object range, the three key point coordinates within the target object range that are relatively fixed and best reflect the target object features are selected by default. In principle, any three key points with obvious features within the target object can be selected. For convenience and typicality, when the target object is a person's face and the anchor point is outside the face range, two eyes and the end of the nose are selected as the three key points by default. Such three key points have significant features, and the triangle formed has good face positioning characteristics, and the obtained anchor point position is more accurate.

[0077] Step S106, generating the stickers according to the display area, tracking area and sticker resources. After determining the setting method of the stickers, setting the position and sticker size information, the stickers will be updated in real time as the face (target object) moves, and the tracking area will always change with the movement of the sticker anchor point to determine the relative position of the sticker and the face (target object). Therefore, when the user takes a photo and adds stickers, the stickers change with the position of the face in the video stream and remain in a specific position relative to the face. For example, if the stickers are cat ears, that is, two stickers form a horizontal mirror image, the cat ear stickers are automatically generated according to the setting method, the adjusted sticker size and the relative position to the face. At the same time, the sticker can be manually adjusted to create personalized requirements.

[0078] Example 2

[0079] like Figure 7 As shown, according to a specific embodiment of the present invention, on the other hand, the present invention provides a sticker generating device 700, comprising:

[0080] The background acquisition unit 701 is used to acquire a background image, and the background image includes a target object. The image in this embodiment can be a static picture, a dynamic picture, or a video stream or a real-time video shot. The image includes a target object that needs to be made into a sticker effect. Usually, the target object can be the face or head of a person or an animal. The image processing device receives an image acquisition instruction from the client, and obtains a background image to be processed by opening and calling an image or camera device. The background image can be a stored picture or video, or a picture or video shot in real time, as a background image for setting stickers.

[0081] The sticker display unit 702 is used to display the display area of ​​the sticker and the anchor point of the sticker in the background image. In the embodiment of the present invention, the background image selects a face, and the sticker effect is actually to paste some pictures on the face. At the same time, these pictures change with the position of the face. The position of the sticker is dynamically adjusted according to the result of face detection, so as to achieve a simple real-time dynamic sticker effect. Among them, the position of the sticker is determined by the anchor point of the sticker. The position of the anchor point on the canvas determines the position of the sticker in the canvas, thereby determining the corresponding position of the sticker and the background image. In this embodiment, the center point of the rectangular area where the sticker is located or one end point of the rectangular area is defined as the anchor point of the sticker. The anchor point coordinates O (x0, y0) are set in the image. The set anchor point can be dragged, and the position can be set in advance according to the type of sticker. The anchor point can be displayed in the 2D drawing board, and the display is represented by a normal black dot, a circle plus a center point graphic "⊙" or other graphic forms, or it can not be displayed. The position of the anchor point is used to locate the position of the sticker.

[0082] The instruction receiving unit 703 is used to receive the sticker import instruction. The position of the sticker on the canvas is set by the user. The sticker can be moved freely within the canvas. The sticker can be dragged and scaled freely within the canvas. According to the obtained sticker anchor point, the sticker setting instruction sent by the client is received, and the initial position of the anchor point of the sticker is set. The initial position is set by the user, that is, the initial coordinates of the anchor point O (x0, y0). The set anchor point can be dragged, and the position can be set in advance according to the type of sticker.

[0083] The sticker importing unit 704 is used to import sticker resources according to the import instruction and display them in the sticker display area. Stickers are imported into the background image, and the stickers are displayed in the target object of the background image according to the setting instruction. For example, cat ear stickers are displayed in a mirror-symmetrical manner on both sides of the mouth, and the cat ear stickers correspond to two rectangular sticker display areas, and the sticker areas are within the scope of the target object. In addition, cat ear stickers are displayed in a mirror-symmetrical manner on both sides of the top of the head, and the cat ear stickers correspond to two rectangular sticker display areas, and the sticker areas are outside the scope of the target object. . Stickers can be varied, such as cat ears, cat whiskers, rabbit ears, horns, hats, decorations, graffiti, etc. The types of stickers include single pictures and sequential frame animations. The display mode of the stickers is selected from one of the display modes of forward display, reverse display, angle display, horizontal mirroring and vertical mirroring. Single selection or multiple selections can be made. Multiple stickers can be set in a symmetrical manner by horizontal or vertical mirroring, or they can be rotated arbitrarily at a certain angle and set at a certain angle. If the background image is a video, it can be set in time sequence. Of course, this setting has a certain degree of freedom and personalization.

[0084] The area selection unit 705 is used to dynamically select a tracking area according to the anchor point position of the sticker, wherein the tracking area is an image area in the target object. The tracking area is used to display the area corresponding to the sticker in the background image. In order to obtain the tracking area, it is first necessary to perform regional segmentation on the face in the background image, and the regional segmentation is performed by segmenting the key points identified on the face, wherein there are at least three key points, usually dozens to hundreds. In this embodiment, the face of a person is set as the target object for analysis, wherein the facial key point information of the face can reflect the position and direction of the face, and the key point information includes internal feature key point information and edge feature key point information. The facial key point detection algorithm in this embodiment integrates the structural information described by the facial edge line into the key point detection, which greatly improves the detection accuracy of the algorithm in extreme cases such as large side faces, exaggerated expressions, occlusion, and blur. In this embodiment, the coordinates of 106 key points of the face are obtained by the face key point detection algorithm. The number of key points is not less than 3 in principle. However, if there are too few key points, the characteristic representation of the target object in the image is too few, and the characteristics of the target object cannot be accurately reflected, and the position of the target object cannot be accurately located. The more the number, the more accurate the characteristic representation of the target object, but the greater the amount of calculation. Usually, dozens to hundreds of them can be selected. The face (target object) in the background image is segmented, including: using the triangulation algorithm of the key point set to segment the face (target object) into N triangular areas, where N is a natural number. The tracking area is a triangular area formed by three key points in the face (target object). When the anchor point of the sticker is inside the area range of the face (target object), the tracking area is the triangular area containing the anchor point; when the anchor point of the sticker is outside the area range of the face (target object), the tracking area is the three key point coordinates within the range of the face (target object) that are relatively fixed and best reflect the characteristics of the target object, that is, two eyes and the end of the nose are selected as the three key points by default. The tracking area is updated in real time according to the position of the anchor point of the sticker in the canvas.

[0085] The sticker generation unit 706 generates the sticker according to the display area, tracking area and sticker resources. After determining the setting method of the sticker, setting the position and sticker size information, the sticker will be updated in real time as the face (target object) moves, and the tracking area will always change with the movement of the sticker's anchor point to determine the relative position of the sticker and the face (target object). When the user takes a photo and adds a sticker, the sticker changes with the position of the face in the video stream and remains in a specific position relative to the face. For example, if the sticker is a cat ear, that is, two stickers form a horizontal mirror image, a cat ear sticker is automatically generated according to the setting method, the adjusted sticker size and the relative position to the face. At the same time, the sticker can be manually adjusted to create personalized requirements.

[0086] Furthermore, the device also includes:

[0087] The anchor point acquisition unit is used to acquire the anchor point of the sticker, and the anchor point is used to locate the position of the sticker. The position of the anchor point of the sticker is set at the center point of the display area of ​​the sticker or a vertex position of the display area. The anchor point coordinates O(x0, y0) are set in the image. The set anchor point can be dragged, and the position can be set in advance according to the type of sticker. This embodiment introduces how to set the relative position of an anchor point coordinate and a target object, and update it in real time. If multiple stickers are set, multiple anchor points need to be set.

[0088] Furthermore, the device also includes:

[0089] The key point acquisition unit is used to acquire the target object on the background image and the key points of the target object, and the key points are used to define the tracking area. The face key point detection algorithm in this embodiment integrates the structural information described by the face edge line into the key point detection, and obtains the coordinates of 106 key points of the face through the face key point detection algorithm. The algorithm obtains the face key point detection dataset WiderFacialLandmarksin-the-wild(WFLW) by capturing the common image feature information of the face, which contains face image data annotated with key points and face attributes, including posture, expression, lighting, makeup, occlusion and blur transformation, aiming to help the academic community to more specifically evaluate the robustness of the key point algorithm under various conditions. In principle, the number of key points should not be less than 3. However, if there are too few key points, the feature representation of the target object in the image will be too small, and the features of the target object cannot be accurately reflected, and the position of the target object cannot be accurately located. The more key points there are, the more accurate the feature representation of the target object will be, but the amount of calculation will also be greater. Usually, dozens to hundreds of key points can be selected. For example, 106 can be selected, or 22, 98, 100, and so on.

[0090] The region segmentation unit is used to perform region segmentation on the target object through the key points. The region segmentation on the face (target object) in the background image includes: using a triangulation algorithm of a key point set to segment the face (target object) into N triangular regions, where N is a natural number. Figure 4 A segmentation diagram of a triangulation algorithm (Delaunay) is shown.

[0091] Furthermore, the device also includes:

[0092] The area selection unit is used to dynamically select the corresponding image area in the target object in real time according to the anchor point of the sticker. The tracking area is determined according to the positional relationship between the anchor point O and the triangle ABC calculated by the above method. When the anchor point is within the target range, the tracking area is determined as the triangular area containing the anchor point, that is, the anchor point O is located inside the triangle ABC, and is updated in real time as the anchor point position changes. When the anchor point position of the sticker is outside the range of the target object (such as a human face), the coordinates of three key points within the range of the target object that are relatively fixed and best reflect the characteristics of the target object are selected by default. In principle, any three key points with obvious features within the target object can be selected. For convenience and typicality, when the target object is a person's face and the anchor point is outside the face range, two eyes and the end of the nose are selected as three key points by default. Such three key points have significant features, and the triangle formed has good face positioning characteristics, and the obtained anchor point position is more accurate.

[0093] Example 3

[0094] This embodiment provides an electronic device, which is used in a method for generating stickers. The electronic device includes: at least one processor; and a memory connected to the at least one processor in communication; wherein:

[0095] The memory stores instructions executable by the one processor, the instructions being executed by the at least one processor to enable the at least one processor to:

[0096] Acquire a background image, wherein the background image includes a target object;

[0097] Displaying a display area of ​​the sticker and an anchor point of the sticker in the background image;

[0098] Receive an import instruction for the sticker;

[0099] Import the resources of the sticker according to the import instruction and display them in the display area of ​​the sticker;

[0100] Dynamically selecting a tracking area according to the anchor point position of the sticker, wherein the tracking area is an image area in the target object; and

[0101] The sticker is generated according to the display area, the tracking area, and resources of the sticker.

[0102] The specific processing method can be found in Example 1.

[0103] Reference below Figure 8, which shows a schematic diagram of the structure of an electronic device 800 suitable for implementing the embodiment of the present disclosure. The electronic device in the embodiment of the present disclosure may include, but is not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 8 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0104] like Figure 8 As shown, the electronic device 800 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage device 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the electronic device 800 are also stored. The processing device 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0105] Typically, the following devices may be connected to the I / O interface 805: input devices 806 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; output devices 807 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 808 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 809. The communication device 809 may allow the electronic device 800 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 8 The electronic device 800 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0106] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 809, or installed from a storage device 808, or installed from a ROM 802. When the computer program is executed by the processing device 801, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0107] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0108] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0109] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device:

[0110] Acquire a background image, which includes a target object; display a display area of ​​a sticker and an anchor point of the sticker in the background image; receive an import instruction for the sticker; import the resources of the sticker according to the import instruction and display them in the display area of ​​the sticker; dynamically select a tracking area according to the anchor point position of the sticker, wherein the tracking area is an image area in the target object; and generate the sticker according to the display area, the tracking area, and the resources of the sticker.

[0111] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0112] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0113] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the name of a unit does not, in some cases, limit the unit itself.

[0114] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

Claims

1. A sticker generation method, characterized in that: include: Acquire a background image, wherein the background image includes a target object; Displaying a display area of ​​the sticker and an anchor point of the sticker in the background image; Receive an import instruction for the sticker; Import the resources of the sticker according to the import instruction and display them in the display area of ​​the sticker; Dynamically selecting a tracking area according to the anchor point position of the sticker, wherein the tracking area is an image area in the target object; as well as Generate the sticker according to the display area, the tracking area and the resources of the sticker; Wherein, when the anchor point of the sticker is inside the area range of the target object, the tracking area is a triangular area containing the anchor point; when the anchor point of the sticker is outside the area range of the target object, the tracking area is a triangular area composed of the coordinates of three fixed key points within the range of the target object; the tracking area is used to determine the relative position of the sticker and the target object; Wherein, the method further comprises: The tracking area is updated in real time according to the position of the anchor point of the sticker in the canvas.

2. The method according to claim 1, characterized in that: The anchor point of the sticker is set at the center point of the display area of ​​the sticker or at a vertex of the display area.

3. The method according to claim 1, characterized in that Also includes: Acquire key points of the target object on the background image; The target object is segmented into regions using the key points.

4. The method according to claim 3, characterized in that The target object has at least three key points.

5. The method according to claim 4, characterized in that The performing region segmentation on the target object comprises: The target object is divided into N triangular regions using a point set triangulation algorithm, where N is a natural number.

6. The method according to claim 5, characterized in that The tracking area is an image area in the target object, specifically a triangular area formed by three key points in the target object corresponding to the anchor point.

7. The method according to claim 1, characterized in that When the target object is a person's face and the anchor point is outside the face range, the three fixed key points are selected as the pupils of the two eyes and the end of the nose, and the tracking area is a triangular area formed by the pupils of the two eyes and the end of the nose.

8. The method according to claim 1, characterized in that The sticker resource includes the sticker type and the sticker display mode.

9. The method according to claim 8, characterized in that The types of stickers include single pictures and sequence frame animations, and the display mode of the stickers is selected from one of forward display, reverse display, angle display, horizontal mirror display and vertical mirror display.

10. The method according to claim 1, characterized in that Also includes: Receive the position adjustment and size scaling instructions of the sticker, so that the sticker can be dragged and scaled arbitrarily within the canvas range.

11. A sticker generating device, characterized in that: include: A background acquisition unit, used to acquire a background image, wherein the background image includes a target object; A sticker display unit, used to display a sticker display area and a sticker anchor point in the background image; An instruction receiving unit, used for receiving an instruction to import the sticker; A sticker importing unit, used to import sticker resources according to the import instruction and display them in the sticker display area; an area selection unit, configured to dynamically select a tracking area according to the anchor point position of the sticker, wherein the tracking area is an image area in the target object; as well as A sticker generating unit, generating the sticker according to the display area, the tracking area and the resources of the sticker; Wherein, when the anchor point of the sticker is inside the area range of the target object, the tracking area is a triangular area containing the anchor point; when the anchor point of the sticker is outside the area range of the target object, the tracking area is a triangular area composed of the coordinates of three fixed key points within the range of the target object; the tracking area is used to determine the relative position of the sticker and the target object; The area selection unit is further used to: update the tracking area in real time according to the position of the anchor point of the sticker in the canvas.

12. The device according to claim 11, characterized in that Also includes: The anchor point acquisition unit is used to acquire the anchor point of the sticker, and the anchor point is used to locate the position of the sticker.

13. The device according to claim 11, characterized in that Also includes: The key point acquisition unit is used to acquire the target object on the background image and the key points of the target object, wherein the key points are used to define the tracking area.

14. The device according to claim 13, characterized in that Also includes: A region segmentation unit is used to perform region segmentation on the target object through the key points, and divide the target object into N triangular regions, where N is a natural number.

15. The device according to claim 11, characterized in that Also includes: The area selection unit is used to dynamically select the corresponding image area in the target object in real time according to the anchor point of the sticker.

16. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 10 is implemented.

17. An electronic device, characterized in that: include: one or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, enables the one or more processors to implement the method according to any one of claims 1 to 10.

Images