A method and device for performing image processing on a specific position of a target object

Abstract

The present invention provides a method and device for image processing of a specific position of a target object, wherein the specific position of the target object involves at least one key point, and the method comprises: collecting a picture containing the target object, training a neural network model of each key point, obtaining a trained neural network model of each key point, and using it to determine whether the corresponding key point is included in the picture; inputting each key point information involved in the specific position of the target object in the target image into the corresponding neural network model respectively, so as to respectively determine whether each key point is in the target image; when all the key points involved are included in the target image, determining to perform image processing on the specific position of the target object. The present invention can accurately perform image processing on each specific position of the target object.

Description

Technical Field

[0001] The present invention relates to the field of Internet information technology, and in particular to a method and device for performing image processing on a specific position of a target object. Background Art

[0002] Based on the various existing image processing technologies on the WeChat mini-program platform, the functions are all based on the in-out mode of processing static images and returning them to the user. The scenes are mostly style changes, image cutting and combination, and background change. The technologies used include image cropping, image synthesis, image tone changes, AI comic style, AI cutout and other technologies.

[0003] In the prior art, when applying stickers to human body parts, a credibility threshold is set for key points of the human body based on empirical values, and stickers can only be applied if the threshold is exceeded.

[0004] Then, when processing images such as stickers, the above method of processing images based on key point thresholds is not accurate because the setting of the empirical value itself is not accurate. Some target objects such as human body parts may not exist but are processed. In other words, the accuracy of image processing based on the existing technology is not high. Summary of the invention

[0005] The object of the present invention is to provide a method and device for performing image processing on specific positions of a target object, which can accurately perform image processing on each specific position of the target object.

[0006] An embodiment of the present invention provides a method for performing image processing on a specific position of a target object, wherein the specific position of the target object involves at least one key point. The method comprises: acquiring a picture containing the target object, training a neural network model for each key point, and obtaining a trained neural network model for each key point, which is used to determine whether the corresponding key point is contained in the picture; inputting information of each key point involved in the specific position of the target object in the target image into the corresponding neural network model respectively, so as to respectively determine whether each key point is in the target image; and when all the key points involved are contained in the target image, determining to perform image processing on the specific position of the target object.

[0007] An embodiment of the present invention also provides a device for performing image processing on a specific position of a target object, wherein the specific position of the target object involves at least one key point, and the device includes: a training module, which collects pictures containing the target object, trains a neural network model for each key point, and obtains a trained neural network model for each key point, which is used to determine whether the corresponding key point is included in the picture; a processing module, which inputs information of each key point involved in the specific position of the target object in the target image into the corresponding neural network model, respectively, to determine whether each key point is in the target image; and a judgment module, which determines to perform image processing on the specific position of the target object when all the key points involved are included in the target image.

[0008] The method for image processing of specific positions of target objects provided by the present invention is based on the credibility of key points returned by the key point detection API, and combines deep learning to train the neural network model of each key point, so as to accurately determine whether the corresponding key point is included in the target image submitted in real time, and realize accurate image processing technology of each specific position of the target object. Thus, the target object image processing technology is more accurate and perfect. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 The present invention is a flowchart of a method for performing image processing on a specific position of a target object according to an embodiment of the present invention.

[0010] Figure 2 The present invention is a schematic diagram of a method for making a human body part sticker according to an embodiment of the present invention.

[0011] Figure 3 The present invention is a schematic diagram of the structure of a device for performing image processing on a specific position of a target object according to an embodiment of the present invention.

[0012] Figure 4 The figure is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention.

[0013] Figure 5 Schematic diagram of sticker effect according to an embodiment of the present invention. DETAILED DESCRIPTION

[0014] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the solutions of the present invention are further described in detail below with reference to the accompanying drawings and embodiments.

[0015] Some existing AI platforms have opened human key point detection APIs, which can accurately estimate the coordinates and credibility of the main key points of the human body in images or videos. The 14 main human key points include: left and right elbows, left and right wrists, left and right shoulders, head, neck, left and right ankles, left and right knees, and left and right hips. It can estimate multiple postures such as standing, sitting, and moving in multiple scenes, thereby realizing the detection and recognition of action postures. Among them, credibility refers to the probability of whether the detected human key points are true.

[0016] Generally, even if there are other interference factors in the picture, such as accidentally entering the frame, sitting, squatting, half-length portrait, selfie, etc., the human key point API will still return the data of 14 key points of the human body. In other words, for example, for a half-length portrait, the human key points of the knees and ankles do not exist, but the coordinates and credibility data of these two key points are still returned, which is inaccurate.

[0017] The present invention uses the key point detection API to process the image to return the key point coordinates and their credibility, and performs image processing on the specific position of the target object in the target image based on the returned credibility. The target image refers to the image to be processed last, and the target image contains the target object, which may contain a single target object or multiple target objects. Among them, the target object includes multiple specific positions, and each target object in the target image may contain all specific positions or some specific positions.

[0018] The present invention provides a flow chart of a method for performing image processing on a specific position of a target object, such as Figure 1 As shown, the specific position of the target object involves at least one key point, and the method includes:

[0019] Step 11: Collect pictures containing the target object, train each key point neural network model, and obtain the trained neural network model of each key point to determine whether the corresponding key point is included in the picture. There are multiple key points, each of which is associated with two other adjacent key points. Taking the first key point as an example, this step specifically includes:

[0020] S111, collect M pictures, and use the credibility of the first key point in each picture, the credibility of two key points associated with the first key point, and the angle between the first key point and the two associated key points as the i-th sample in the deep learning training set, i∈M, M is a natural number; each of the collected pictures includes all specific positions of the target object, or part of the specific positions of the target object;

[0021] S112, inputting the information of the i-th sample into the first key point neural network model to obtain a prediction value of whether the first key point is included in the image;

[0022] S113. According to the predicted value and the true value of whether the i-th sample is included in the picture, a loss function is constructed, the loss function is optimized by taking the minimum value, the network weight parameters are updated, and the trained first key point neural network model is obtained.

[0023] Step 12: Input each key point information related to the specific position of the target object in the target image into the corresponding neural network model to determine whether each key point is in the target image.

[0024] When the key points involved include the first key point, taking the first key point as an example, inputting the first key point information into the first neural network model to determine whether the first key point is in the target image, specifically including:

[0025] The credibility of the first key point, the credibility of the two key points associated with the first key point, and the angle between the first key point and the two associated key points are input into the trained first key point neural network model to obtain whether the first key point is included in the target image.

[0026] Step 13: When all the key points involved are included in the target image, determine to perform image processing on the specific position of the target object.

[0027] At this point, the method of the present invention for performing image processing on a specific position of a target object is completed. Among them, stickers, or blurring, mosaicing, enlarging, reducing, flipping a certain angle, mirroring, deforming, etc., can all be used as an implementation method in image processing. Since the training set contains a large number of various target object pictures, various situations of the target object posture can be taken into account, so the key point neural network model can obtain a more accurate probability of whether the key point is included in the picture after training. Therefore, it can more accurately determine whether the key point is actually included in the picture, thereby achieving more accurate image processing.

[0028] The following is an example of the method of performing image processing on a specific position of a target object of the present invention, using the key points of the human body as an example. The specific image processing method is a sticker method, and the target object is a human body. The human body has 14 main key points and 12 parts. The specific position of the target object refers to a part of the human body. In the present invention, the target object can be a human body, or other objects with key points, and the coordinates and credibility of the key points can be detected.

[0029] Specifically, there are 14 key points for the human body, including left and right elbows, left and right wrists, left and right shoulders, head, neck, left and right ankles, left and right knees, and left and right hips. To cover sticker images of various parts of the human body in the original image, there are 12 main parts of the human body, so a set of sticker images with human skeletons is provided in advance, including 12 small sticker images, each small sticker image corresponds to a human body part: head, body, left upper arm, right upper arm, left forearm, right forearm, left hand, right hand, left thigh, right thigh, left calf, and right calf. Each small sticker image has its own length H and width W.

[0030] The embodiment of the present invention is built on the human key point detection API, and its application scenarios and extended applications. In a specific WeChat applet scenario, the key point detection API is used to process the 14 key points of the human body and their credibility returned from the static image, and the 12 human body parts of the characters in the picture are filtered based on the returned credibility. Among them, the filtering of the 12 human body parts of the characters in the picture based on the returned credibility is specifically through deep learning, training each human key point neural network model, and obtaining each human key point neural network model after training; each human key point information involved in the human body part in the original picture to be pasted is input into the corresponding neural network model respectively, so as to determine whether each human key point is in the original picture. Only when all the human key points involved in the human body part are included in the original picture, it is determined to paste the human body part. The character in the picture can be a single character or multiple characters, and can be a full-body photo or a half-body photo, without limitation. In this way, the present invention realizes accurate mapping of various parts of the human body. The present invention can achieve the goal of not pasting back figures, not pasting side figures, not pasting people who are mistakenly in the frame, and not pasting blurred figures, and then accurately, coherently, and overall coordinated without the problems of wrong pasting, erroneous pasting, and incoherence.

[0031] The present invention provides a schematic flow chart of a method for attaching a human body part sticker, such as Figure 2 As shown, this embodiment uses a deep learning method to train a neural network model, but the method of training a neural network model includes but is not limited to this, and any other learning method that can obtain a trained neural network model for determining whether a key point is included in a picture is included in the solution of the present invention. The method includes the following steps:

[0032] Step 21: Train each human key point neural network model through deep learning to obtain a trained neural network model of each human key point, which is used to determine whether the corresponding human key point is included in the picture.

[0033] There are multiple human body key points, such as 14 main key points, which are called the first human body key point to the fourteenth human body key point.

[0034] Through deep learning, the first human key point neural network model is trained to obtain the trained first human key point neural network model, which specifically includes:

[0035] S211, collect M pictures, and use the credibility of the first human key point in each picture, the credibility of two human key points associated with the first human key point, and the angle between the first human key point and the two associated human key points as the i-th sample in the deep learning training set, i∈M, M is a natural number; the M pictures include full-body single-person pictures and half-body single-person pictures;

[0036] The more pictures included in the training set, the more accurate the training results. The pictures of the present invention include full-body single-person pictures and half-body single-person pictures. Half-body single-person pictures can be used on the existing training set to process the upper body, above the knees, or remove some joints. Use PS to process part of the training set, add sunlight, add shadows, etc. In this way, the training set of the present invention now includes complete people and incomplete people and character shadows.

[0037] However, it should be noted that even for a half-body single-person image, the coordinates and credibility of the missing human key points can be obtained through the human key point detection API.

[0038] S212, adopting a deep learning method, using the deep learning training set to train the first human key point neural network model, to obtain a trained first human key point neural network model. The training process includes steps S2121 and S2122.

[0039] Each human key point corresponds to a human key point neural network model, and each human key point neural network model corresponds to a set of optimal network weight parameters after training. For example, the first human key point corresponds to the first human key point neural network model. The trained first human key point neural network model corresponds to a set of optimal network weight parameters.

[0040] S2121, inputting the information of the i-th sample into the first human key point neural network model to obtain a prediction value of whether the first human key point is included in the picture;

[0041] S2122. According to the predicted value and the true value of whether the i-th sample is included in the picture, a loss function is constructed, the minimum value of the loss function is optimized, the network weight parameters are updated, and the trained first human key point neural network model is obtained.

[0042] Whether each sample human key point is included in the true value of the image can be determined in advance by observing the sample image. By observing the sample image, if a human key point is located in the image, the true value of the human key point, that is, the true value of the sample, is set to 1. If a human key point is missing and not included in the image, the true value of the human key point is set to 0.

[0043] When the predicted value of the sample is closer to the true value, the loss function takes the minimum value. At this time, the network weight parameters are optimal and the model training is completed. The trained neural network model can be used in the subsequent step 22 to predict the credibility of the key points of the human body at the same position in the original image.

[0044] Step 22: input each key point information of the human body part in the original image to be pasted into the corresponding neural network model to determine whether each key point of the human body is in the original image.

[0045] As mentioned above, human body parts cover multiple human body key points, so some human body parts, such as the body, involve four human body key points: left shoulder, right shoulder, left hip and right hip, and some human body parts, such as the left calf, involve two human body key points: left knee and left ankle. Some human body parts, such as the left hand, involve one human body key point, the left elbow, because the human body key point API cannot return the hand points in the data returned, only the wrist points.

[0046] When the human body part in the original image to be pasted involves two human body key points: the first human body key point and the second human body key point,

[0047] Input the credibility of the first human key point, the credibility of two human key points associated with the first human key point, and the angle between the first human key point and the two associated human key points into the trained first human key point neural network model to obtain whether the first human key point is included in the original image;

[0048] The credibility of the second human body key point, the credibility of the two human body key points associated with the second human body key point, and the angle between the second human body key point and the two associated human body key points are input into the trained second human body key point neural network model to obtain whether the second human body key point is included in the original image.

[0049] Step 23: When all the key points of the human body involved are included in the original image, determine to apply stickers to the human body parts.

[0050] At this point, the human body part sticker method of the present invention is completed. Since the training set contains a large number of various human pictures, various situations of human postures can be taken into account, so the human body key point neural network model can obtain a more accurate probability of whether the human body key point is included in the picture after training. Therefore, it can be more accurately determined whether the human body key point is actually included in the picture, so that more accurate stickering can be achieved.

[0051] To clearly explain the present invention, the specific implementation is described in detail below. For example, for the first human key point neural network model, the input information for training the first human key point neural network model is the credibility of the first human key point, the credibility of the two human key points associated with the first human key point, and the angle between the first human key point and the two associated human key points, and the above information is used as a sample i in the deep learning training set M. If the first human key point is associated with the second human key point and the third human key point, the training process is as follows:

[0052] Step 31, collect M pictures, and form a four-dimensional vector (a, b, c, A) with the credibility a of the first human key point, the credibility b of the second human key point, the credibility c of the third human key point, and the angle A between the first human key point and the second and third human key points in each picture as the i-th sample in the training set. A can be further normalized to obtain a value between (0, 1).

[0053] It should be noted that, in order to improve the accuracy of the sticker, the present invention needs to associate the first human key point with two human key points and give the angle between the two associated human key points, so a four-dimensional vector is formed. Therefore, if the first human key point is only associated with the second human key point, the credibility value of the third human key point can be set to 1.

[0054] Step 32: Input the information of the i-th sample into the first human key point neural network model to obtain a prediction value h of whether the first human key point is included in the image. θ (x)=θ0+θ1x1+θ2x2+θ3x3+θ4x4;

[0055] Among them, θ represents the network weight parameter, x represents the neuron, and n represents the information feature dimension. In the present invention, it is a four-dimensional vector, so n=4, and x1 to x4 correspond to the four-dimensional vector (a, b, c, A) respectively.

[0056] Step 33: Construct a loss function based on the predicted value and true value of whether the i-th sample is included in the image The loss function is minimized and optimized, and the network weight parameter θ is updated to obtain the first human key point neural network model after training.(i) Indicates the true value of whether the i-th sample of the first human key point is included in the image. This value can be determined by naked eye observation.

[0057] The first human key point neural network model after training corresponds to a set of optimal network weight parameters θ0 to θ4. Each human key point obtains a trained neural network model. Therefore, the optimal θ0 to θ4 obtained by each human key point neural network model after training is different.

[0058] Among them, the gradient descent method is used to calculate the gradient of the loss function, and the direction with the fastest gradient descent is selected to minimize the loss function.

[0059] At this point, the training process of the neural network model for each key point of the human body can be clearly explained.

[0060] Then, when an original image is submitted in real time, in order to determine whether to apply stickers to the various human body parts in the original image, each human body part needs to be credibility filtered through a neural network model. Only when all the key points of the human body involved in the human body part are included in the original image, it is determined that the human body part will be applied with stickers.

[0061] Therefore, it is necessary to input the key point information of each human body part involved in the human body part into the corresponding neural network model for prediction to determine whether it is in the original image.

[0062] For example, assuming that the human body part involves the first human body key point, the human body key point detection API is used to obtain (a, b, c, A) corresponding to the first human body key point, and the four-dimensional vector (a, b, c, A) is input into the trained first human body key point neural network model to obtain the value of whether the first human body key point is included in the original image. This value is generally a floating point value. According to experience, values ​​greater than 0.5 are generally set to 1, indicating that the first human body key point is included in the original image. Values ​​less than 0.5 are set to 0, indicating that the first human body key point is not included in the original image.

[0063] The four-dimensional vector (a, b, c, A) is input into the trained first human key point neural network model h θ (x)=θ0+θ1x1+θ2x2+θ3x3+θ4x4. From this formula, it can be seen that x1 to x4 correspond to four-dimensional vectors (a, b, c, A) respectively, and θ0 to θ4 are also determined values ​​after optimization. Therefore, the h value can be determined, that is, whether the first human body key point is included in the original image.

[0064] The following is a detailed description of the sticker scheme for each part of the human body.

[0065] Embodiment 1

[0066] The principles of the left and right upper arms sticker method are the same, so this embodiment takes the left upper arm as an example for description.

[0067] When the original human body part is the left upper arm, the human body key points involved are the left elbow and the left shoulder. The two human body key points associated with the left elbow are the left shoulder and the left wrist; the human body key points associated with the left shoulder are the neck and the left elbow.

[0068] For the left elbow, a key point of the human body, the credibility of the left elbow, the credibility of the left shoulder, the credibility of the left wrist, and the angle between the left elbow, the left shoulder and the left wrist are input into the trained neural network model corresponding to the left elbow to determine whether the left elbow is included in the original image. When the h value is greater than 0.5, the value is set to 1, indicating that the left elbow is included in the original image; when the h value is less than 0.5, the value is set to 0, indicating that the left elbow is not included in the original image.

[0069] For the left shoulder, which is a key point of the human body, the credibility of the left shoulder, the credibility of the left elbow, the credibility of the neck, and the angle between the left shoulder, the left elbow and the neck are input into the trained neural network model corresponding to the left shoulder to determine whether the left shoulder is included in the original image.

[0070] When the values ​​of the left elbow and the left shoulder in the original image are both 1, it is determined that the left upper arm is to be pasted.

[0071] The following is a brief description of how to apply stickers to various human body parts in a specific application. When the human body part is the left upper arm, the preset left upper arm sticker image has a length of H1 and a width of W1.

[0072] According to the coordinates of the left shoulder and the left elbow in the original image, the length h1 of the left upper arm in the original image and the rotation angle A1 of the left upper arm sticker image on the original image are obtained;

[0073] Compare the length h1 of the left upper arm in the original image with the length H1 of the left upper arm sticker image, and obtain the scaling ratio P1=h1 / H1 of the left upper arm sticker image;

[0074] According to the left upper arm sticker image scaling ratio P1, the scaled left upper arm sticker image width w1 is obtained;

[0075] The preset left upper arm sticker image is scaled according to h1 and w1 to obtain a scaled left upper arm sticker image;

[0076] Calculate the coordinates (X1, Y1) of the position where the scaled left upper arm sticker image needs to be placed on the original image based on the coordinates of the left shoulder of the original image;

[0077] Place the upper left corner of the scaled left upper arm sticker image at (X1, Y1);

[0078] The scaled left upper arm sticker image is rotated according to the rotation angle A1.

[0079] To clearly explain the above application, the following is a detailed description. According to the original image uploaded by the user, the API interface is called to obtain the data of 14 key points of the human body, including the coordinates of each key point and the credibility of the key point. This includes obtaining the coordinates of the left shoulder (x1, y1) and the coordinates of the left elbow. And it is known that the length of the left upper arm sticker image is H1 and the width is W1.

[0080] 1) The distance between the left shoulder and the left elbow is calculated based on the coordinates of the left shoulder and the left elbow, that is, the length h1 of the left upper arm in the original image; the angle between the left upper arm and the vertical direction can also be calculated based on the coordinates of the left shoulder and the left elbow, that is, the rotation angle A1 of the left upper arm sticker image on the original image.

[0081] 2) Compare the length h1 of the left upper arm in the original image with the length H1 of the left upper arm sticker image, and obtain the scaling ratio P1=h1 / H1 of the left upper arm sticker image;

[0082] It is known that the width of the left upper arm sticker image is W1, so according to w1=P1*W1, the scaled width w1 of the left upper arm sticker image is obtained.

[0083] 3) Scale the length of the left upper arm sticker image from H1 to h1, and the width from W1 to w1.

[0084] 4) Calculate the coordinates (X1, Y1) of the position where the scaled left upper arm sticker image needs to be placed on the original image: Based on A1, if A1 is an acute angle, then

[0085] If A1 is an obtuse angle, then

[0086] 5) Since the sticker is an image, place the upper left corner of the scaled left upper arm sticker image at (X1, Y1).

[0087] 6) The scaled left upper arm sticker image is rotated based on the center point between the upper left corner and the upper right corner of the scaled left upper arm sticker image according to the rotation angle A1.

[0088] Due to engineering requirements, the front-end drawing rotation angle A1 is required to be 0-360, which is handled by the back-end when processing data.

[0089] In this way, the left upper arm sticker image has simply covered the left upper arm of the original image.

[0090] Embodiment 2

[0091] The principles of the left and right forearm sticker methods are the same, so this embodiment is described using the left forearm as an example.

[0092] When the original human body part is the left forearm, the human body key points involved are the left wrist and the left elbow. The two human body key points associated with the left elbow are the left wrist and the left shoulder; the human body key point associated with the left wrist is the left elbow. The credibility value of the other human body key point associated with the left wrist is set to 1.

[0093] For the left elbow, a key point of the human body, the credibility of the left elbow, the credibility of the left wrist, the credibility of the left shoulder, and the angle between the left elbow, the left wrist and the left shoulder are input into the trained neural network model corresponding to the left elbow to determine whether the left elbow is included in the original image. When the h value is greater than 0.5, the value is set to 1, indicating that the left elbow is included in the original image; when the h value is less than 0.5, the value is set to 0, indicating that the left elbow is not included in the original image.

[0094] For the left wrist, which is a key point of the human body, the credibility of the left wrist, the credibility value 1, and the angle between the left wrist and the left elbow and the vertical direction are input into the trained neural network model corresponding to the left wrist to determine whether the left wrist is included in the original image.

[0095] When the values ​​of the left wrist and the left elbow in the original image are both 1, it is determined that the left forearm is to be pasted.

[0096] The following is a brief description of how to apply stickers to various human body parts in a specific application. When the human body part is the left forearm, the preset left forearm sticker image has a length of H2 and a width of W2.

[0097] According to the coordinates of the left elbow and the left wrist in the original image, the length h2 of the left forearm in the original image and the rotation angle A2 of the left forearm sticker image on the original image are obtained;

[0098] The preset left forearm sticker image is scaled according to h2 and the left upper arm w1 to obtain a scaled left forearm sticker image;

[0099] Calculate the coordinates (X2, Y2) of the position where the scaled left forearm sticker image needs to be placed on the original image based on the coordinates of the left elbow in the original image;

[0100] Place the upper left corner of the scaled left forearm sticker image at (X2, Y2);

[0101] The scaled left forearm sticker image is rotated according to the rotation angle A2.

[0102] To clearly illustrate the above application, the following is a detailed description. Still based on the original image uploaded by the user, the coordinates of the left elbow (x2, y2) and the coordinates of the left wrist are obtained. And it is known that the length of the left forearm sticker image is H2 and the width is W2.

[0103] 1) The distance between the left elbow and the left wrist is calculated based on the coordinates of the left elbow and the left wrist, that is, the length h2 of the left forearm in the original image; the angle between the left forearm and the vertical direction can also be calculated based on the coordinates of the left elbow and the left wrist, that is, the rotation angle A2 of the left forearm sticker image on the original image.

[0104] 2) During the display of the image, the upper arm and the forearm may appear to be "folded" or very short visually, and according to the calculation of key points, a "short and thick" effect will appear. In order to avoid this during the display process, the width of the upper and lower arms is kept the same. That is, the width of the scaled left forearm sticker image is equal to the left upper arm w1.

[0105] 3) Scale the length of the left forearm sticker image from H2 to h2, and the width from W2 to w1.

[0106] 4) Calculate the coordinates (X2, Y2) of the position where the scaled left forearm sticker image needs to be placed on the original image: Based on A2, if A2 is an acute angle, then

[0107] If A2 is an obtuse angle, then

[0108] 5) Since the sticker is an image, place the upper left corner of the scaled left forearm sticker image at (X2, Y2).

[0109] 6) Rotate the scaled left forearm sticker image about the center point between the upper left corner and the upper right corner of the scaled left forearm sticker image according to the rotation angle A2.

[0110] Due to engineering requirements, the front-end drawing rotation angle A2 is required to be 0-360, which is handled by the back-end when processing data.

[0111] Embodiment 3

[0112] The principles of the left-hand and right-hand sticker methods are the same, so this embodiment is described using the left hand as an example.

[0113] When the human body part in the original image is the left hand, since the human body key point API cannot return the hand points in the data returned, only the wrist points are received, so the human body key point involved is the left elbow.

[0114] For the left elbow, a key point of the human body, the credibility of the left elbow, the credibility of the left wrist, the credibility of the left shoulder, and the angle between the left elbow, the left wrist and the left shoulder are input into the trained neural network model corresponding to the left elbow to determine whether the left elbow is included in the original image. When the h value is greater than 0.5, the value is set to 1, indicating that the left elbow is included in the original image; when the h value is less than 0.5, the value is set to 0, indicating that the left elbow is not included in the original image.

[0115] Therefore, we only need to judge that when the value of the left elbow in the original image is 1, we can decide to apply the sticker to the left hand.

[0116] The following is a brief description of how to apply stickers to various human body parts in a specific application. When the human body part is the left hand, the preset left hand sticker image has a length of H3 and a width of W3.

[0117] According to the scaling ratio of the left forearm P2=h2 / H2, the preset left hand sticker image is scaled to obtain a scaled left hand sticker image;

[0118] Calculate the coordinates (X3, Y3) of the position where the scaled left-hand sticker image needs to be placed on the original image based on the coordinates of the left wrist in the original image;

[0119] Place the upper left corner of the scaled left-hand sticker image at (X3, Y3);

[0120] The scaled left hand sticker image is rotated according to the left forearm rotation angle A2.

[0121] To clearly explain the above application, a detailed description is given below. Still based on the original image uploaded by the user, and it is known that the length of the left-hand sticker image is H3 and the width is W3.

[0122] 1) According to the scaling ratio of the left forearm P2=h2 / H2, the length of the left hand sticker image is scaled from H3 to h3, and the width is scaled from W3 to w3.

[0123] 2) Calculate the coordinates (X3, Y3) of the position where the scaled left-hand sticker image needs to be placed on the original image: Among them, (x3, y3) is the coordinate of the left wrist in the original image, and w3 is the width of the sticker image after the left hand is scaled.

[0124] 3) Since the sticker is an image, place the upper left corner of the scaled left-hand sticker image at (X3, Y3).

[0125] It should be noted that the w3 and h3 of the hand calculated based on the forearm may have many special cases, and the length-width ratio of the hand is incorrect. If the difference ratio between the left and right hand widths is greater than 1.2 (the ideal value after multiple tests), the left and right hands have the same w3 and h3 (the w3 and h3 of the larger hand). In addition, the calculated w3 and h3 of the hand are slightly larger than the actual hand size, which can cover the hand and have a better display effect.

[0126] 4) Since the human body key point API does not return the hand points, only the wrist points are returned; that is, the hand part uses the same forearm angle. Therefore, the left hand rotation angle is the same as the left forearm rotation angle, and the scaled left hand sticker image is rotated according to the rotation angle A2.

[0127] Embodiment 4

[0128] When the human body part in the original image is the head, the human body key points involved are the head center point and the neck. The human body key points associated with the neck are the head center point and the left shoulder. The human body key point associated with the head center point is the neck, and the credibility value of another human body key point associated with the head center point is set to 1.

[0129] For the key point neck of the human body, the credibility of the neck, the credibility of the head center, the credibility of the left shoulder, and the angle between the neck, the head center and the left shoulder are input into the trained neural network model corresponding to the neck to determine whether the neck is included in the original image. When the h value is greater than 0.5, the value is set to 1, indicating that the neck is included in the original image; when the h value is less than 0.5, the value is set to 0, indicating that the neck is not included in the original image.

[0130] For the key point of the human body, the credibility of the head center point, the credibility of the neck, the credibility value 1, and the angle between the head center point and the neck and the vertical direction are input into the trained neural network model corresponding to the head center point to determine whether the head center point is included in the original image.

[0131] When the values ​​of the center points of the neck and head in the original image are both 1, it is determined that the sticker is applied to the head.

[0132] The following is a brief description of how to apply stickers to various human body parts in a specific application. When the human body part is the head, the preset head sticker image length is H4 and width is W4.

[0133] According to the coordinates of the head and neck in the original image, the length h4 of the head in the original image and the rotation angle A4 of the head sticker image on the original image are obtained;

[0134] Compare the head length h4 in the original image with the length H4 of the head sticker image, and obtain the scaling ratio of the head sticker image P4=h4 / H4;

[0135] According to the scaling ratio P4 of the head sticker image, the scaled head sticker image width w4 is obtained;

[0136] The preset head sticker image is scaled according to h4 and w4 to obtain a scaled head sticker image;

[0137] Calculate the coordinates (X4, Y4) of the position where the scaled head sticker image needs to be placed on the original image based on the coordinates of the original image head;

[0138] Place the upper left corner of the scaled head sticker image at (X4, Y4);

[0139] According to the rotation angle Rotate the scaled head sticker image.

[0140] To clearly explain the above application, the following is a detailed description. Still based on the original image uploaded by the user, the coordinates of the center point of the head (x4, y4) and the coordinates of the neck are obtained. And it is known that the length of the head sticker image is H4 and the width is W4.

[0141] 1) The distance from the center of the head to the neck in the original image is calculated based on the coordinates of the head and neck. According to the normal proportion of the human body and multiple tests, the length of the head in the original image, h4, is equal to twice the distance from the center of the head to the neck in the original image. The rotation angle A4 of the head sticker image on the original image can also be calculated based on the coordinates of the head and neck.

[0142] 2) Compare the head length h4 in the original image with the head sticker image length H4 to obtain the head sticker image scaling ratio P4=h4 / H4;

[0143] It is known that the width of the head sticker image is W4, so according to w4=P4*W4, the scaled head sticker image width w4 is obtained.

[0144] 3) Scale the length of the head sticker image from H4 to h4 and the width from W4 to w4.

[0145] 4) Calculate the coordinates (X4, Y4) of the position where the scaled head sticker image needs to be placed on the original image.

[0146] Specifically:

[0147] 5) Since the sticker is an image, place the upper left corner of the scaled head sticker image at (X4, Y4).

[0148] 6) A4_actual = A4_calculated / 2: Since the angle is calculated from the center of the head, not the top of the head, if it is not divided by 2, the rotation angle of the head will be too large. The return point is the center point, so the angle divided by 2 is exactly the rotation angle of the head.

[0149] There are two types of head rotation: left or right, which is determined by the size of the horizontal coordinates of the head and neck.

[0150] Currently, there are multiple heads: the scaling can be customized according to each head, and the data can be returned. The function of shaking the sticker to change the head is realized in the sticker map, so the head sticker is customized, so that no matter what kind of head sticker, it can be accurately pasted on the center point of the head, meeting the diversification of head stickers and the scalability of applications.

[0151] Embodiment 5

[0152] When the human body part of the original image is the body, the human body key points involved are the left shoulder, right shoulder, left hip and right hip. The human body key points associated with the left shoulder are the left elbow and neck. The human body key points associated with the left hip are the left knee and neck. The human body key points associated with the right shoulder are the right elbow and neck. The human body key points associated with the right hip are the right knee and neck.

[0153] For the left shoulder, which is a key point of the human body, the credibility of the left shoulder, the credibility of the left elbow, the credibility of the neck, and the angle between the left shoulder, the left elbow and the neck are input into the trained neural network model corresponding to the left shoulder to determine whether the left shoulder is included in the original image.

[0154] For the left hip, which is a key point of the human body, the credibility of the left hip, the credibility of the left knee, the credibility of the neck, and the angle between the left hip, the left knee and the neck are input into the trained neural network model corresponding to the left hip to determine whether the left hip is included in the original image.

[0155] For the right shoulder, which is a key point of the human body, the credibility of the right shoulder, the credibility of the right elbow, the credibility of the neck, and the angle between the right shoulder, the right elbow and the neck are input into the trained neural network model corresponding to the right shoulder to determine whether the right shoulder is included in the original image.

[0156] For the right hip, a key point of the human body, the credibility of the right hip, the credibility of the right knee, the credibility of the neck, and the angle between the right hip, the right knee and the neck are input into the trained neural network model corresponding to the right hip to determine whether the right hip is included in the original image.

[0157] When the values ​​of the left shoulder, right shoulder, left hip and right hip in the original image are all 1, it is determined that the body is pasted.

[0158] The following is a brief description of how to apply stickers to various human body parts in a specific application. When the human body part is the body, the preset body sticker image has a length of H5 and a width of W5.

[0159] According to the coordinates of the center points of the neck and left and right hips in the original image, the length h5 of the body in the original image and the rotation angle A5 of the body sticker image on the original image are obtained;

[0160] According to the coordinates of the left shoulder and the right shoulder in the original image, the width w5 of the body in the original image is obtained;

[0161] The preset body sticker image is scaled according to h5 and w5 to obtain a scaled body sticker image;

[0162] Calculate the coordinates (X5, Y5) of the position where the scaled body sticker image needs to be placed on the original image based on the coordinates of the right shoulder of the original image;

[0163] Place the upper left corner of the scaled body sticker image at (X5, Y5);

[0164] Rotate the scaled body sticker image according to the rotation angle A5.

[0165] To clearly explain the above application, the following is a detailed description. Still based on the original image uploaded by the user, the coordinates of the neck and the left and right hips are obtained. And it is known that the length of the body sticker image is H5 and the width is W5.

[0166] 1) According to the coordinates of the center points of the neck and left and right hips in the original image, the length h5 of the body in the original image and the rotation angle A5 of the body sticker image on the original image are obtained.

[0167] The width w5 of the body in the original image is obtained according to the coordinates of the left shoulder and the right shoulder (x5, y5) in the original image.

[0168] 2) Scale the length of the body sticker image from H5 to h5 and the width from W5 to w5.

[0169] 3) Calculate the position coordinates (X5, Y5) where the scaled body sticker image needs to be placed on the original image based on the coordinates of the right shoulder of the original image; specifically, X5=x5; Y5=y5.

[0170] 4) Place the upper left corner of the scaled body sticker image at the coordinate position of the right shoulder of the original image.

[0171] 6) Rotate the scaled body sticker image at the center point between the upper left corner and the upper right corner of the scaled body sticker image according to the rotation angle A5.

[0172] There are two cases of body rotation: left or right. In some special cases, the half body and hip points cannot be accurately identified, and when the hip point reliability is <0.2, A5=0

[0173] Embodiment 6

[0174] The principles of the left thigh and right thigh sticker method are the same, so this embodiment is described using the left thigh as an example.

[0175] When the human body part of the original image is the left thigh, the human body key points involved are the left knee and the left hip. The two human body key points associated with the left knee are the left hip and the left ankle; the human body key points associated with the left hip are the neck and the left knee.

[0176] For the left knee, a key point of the human body, the credibility of the left knee, the credibility of the left hip, the credibility of the left ankle, and the angle between the left knee, the left hip and the left ankle are input into the trained neural network model corresponding to the left knee to determine whether the left knee is included in the original image. When the h value is greater than 0.5, the value is set to 1, indicating that the left knee is included in the original image; when the h value is less than 0.5, the value is set to 0, indicating that the left knee is not included in the original image.

[0177] For the left hip, which is a key point of the human body, the credibility of the left hip, the credibility of the left knee, the credibility of the neck, and the angle between the left hip, the left knee and the neck are input into the trained neural network model corresponding to the left hip to determine whether the left hip is included in the original image.

[0178] When the values ​​of the left knee and the left hip in the original image are both 1, it is determined that the left thigh is to be pasted.

[0179] The following is a brief description of how to apply stickers to various parts of the human body in a specific application. When the human body part is the left thigh, the preset left thigh sticker image has a length of H6 and a width of W6.

[0180] According to the coordinates of the left hip and left knee in the original image, the length h6 of the left thigh in the original image and the rotation angle A6 of the left thigh sticker image on the original image are obtained;

[0181] Compare the length h6 of the left thigh in the original image with the length H6 of the left thigh sticker image, and obtain the scaling ratio P6=h6 / H6 of the left thigh sticker image;

[0182] According to the left thigh sticker image scaling ratio P6, the scaled left thigh sticker image width w6 is obtained;

[0183] The preset left thigh sticker image is scaled according to h6 and w6 to obtain a scaled left thigh sticker image;

[0184] Calculate the coordinates of the position (X6, Y6) where the scaled left thigh sticker image needs to be placed on the original image based on the coordinates of the left hip of the original image.

[0185] Place the upper left corner of the scaled left thigh sticker image at (X6, Y6);

[0186] Rotate the scaled left thigh sticker image according to the rotation angle A6.

[0187] Embodiment 7

[0188] The principles of the left calf and right calf sticker methods are the same, so this embodiment is described using the left calf as an example.

[0189] When the original human body part is the left calf, the human body key points involved are the left knee and the left ankle. The two human body key points associated with the left knee are the left hip and the left ankle; the human body key point associated with the left ankle is the left knee. The credibility value of the other human body key point associated with the left ankle is set to 1.

[0190] For the left knee, a key point of the human body, the credibility of the left knee, the credibility of the left hip, the credibility of the left ankle, and the angle between the left knee, the left hip and the left ankle are input into the trained neural network model corresponding to the left knee to determine whether the left knee is included in the original image. When the h value is greater than 0.5, the value is set to 1, indicating that the left knee is included in the original image; when the h value is less than 0.5, the value is set to 0, indicating that the left knee is not included in the original image.

[0191] For the left ankle, which is a key point of the human body, the credibility of the left ankle, the credibility value 1, and the angle between the left ankle and the left knee and the vertical direction are input into the trained neural network model corresponding to the left ankle to determine whether the left ankle is included in the original image.

[0192] When the values ​​of the left knee and the left ankle in the original image are both 1, it is determined that the left calf is to be pasted.

[0193] The following is a brief description of how to apply stickers to various human body parts in a specific application. When the human body part is the left calf, the preset left calf sticker image has a length of H7 and a width of W7.

[0194] According to the coordinates of the left knee and left ankle in the original image, the length h7 of the left calf in the original image and the rotation angle A7 of the left calf sticker image on the original image are obtained;

[0195] Compare the length h7 of the left calf in the original image with the length H7 of the left calf sticker image, and obtain the scaling ratio P7=h7 / H7 of the left calf sticker image;

[0196] According to the left calf sticker image scaling ratio P7, the scaled left calf sticker image width w7 is obtained;

[0197] The preset left calf sticker image is scaled according to h7 and w7 to obtain a scaled left calf sticker image;

[0198] Calculate the coordinates of the scaled left calf sticker image on the original image based on the coordinates of the left knee in the original image (X7, Y7);

[0199] Place the upper left corner of the scaled left calf sticker image at (X7, Y7);

[0200] The scaled left calf sticker image is rotated according to the rotation angle A7.

[0201] According to the above-mentioned embodiments, the credibility of the key points of the human body involved in each human body part can be accurately filtered, so that mapping can be achieved more accurately.

[0202] Based on the same inventive concept, the present invention also provides a device for performing image processing on a specific position of a target object, wherein the specific position of the target object involves at least one key point, and the structural schematic diagram is shown in FIG. Figure 3 As shown, the device comprises:

[0203] The training module 301 collects pictures containing target objects, trains each key point neural network model, and obtains each key point neural network model after training, which is used to determine whether the corresponding key point is included in the picture;

[0204] The processing module 302 inputs each key point information related to the specific position of the target object in the target image into the corresponding neural network model to determine whether each key point is in the target image;

[0205] The judgment module 303 determines to perform image processing on a specific position of the target object when all the key points involved are included in the target image.

[0206] There are multiple key points, each of which is associated with two other adjacent key points. The training module 301 collects pictures containing the target object, trains each key point neural network model, and obtains each key point neural network model after training. When the key point is the first key point, it is specifically used for:

[0207] M pictures are collected, and the credibility of the first key point in each picture, the credibility of the two key points associated with the first key point, and the angle between the first key point and the two associated key points are used as the i-th sample in the deep learning training set, i∈M, M is a natural number; each collected picture includes all specific positions of the target object, or part of the specific positions of the target object;

[0208] Input the information of the i-th sample into the first key point neural network model to obtain a prediction value of whether the first key point is included in the image;

[0209] According to the predicted value and the true value of whether the i-th sample is contained in the picture, a loss function is constructed, the loss function is minimized for optimization, the network weight parameters are updated, and the first key point neural network model after training is obtained.

[0210] The processing module 302 inputs each key point information related to the specific position of the target object in the target image into the corresponding neural network model to determine whether each key point is in the target image. When the key points involved include the first key point, it is specifically used to:

[0211] The credibility of the first key point, the credibility of the two key points associated with the first key point, and the angle between the first key point and the two associated key points are input into the trained first key point neural network model to obtain whether the first key point is included in the target image.

[0212] The image processing includes performing at least one of the following operations on a specific position of the target object: pasting, blurring, mosaicing, zooming in, zooming out, flipping a certain angle, mirroring, and deformation.

[0213] The modules of the above embodiments may be integrated into one or deployed separately; they may be combined into one module or further divided into multiple sub-modules.

[0214] In addition, an electronic device is also provided in the embodiment of the present application, and the structural diagram is as follows Figure 4 As shown, it includes a memory 401, a processor 402, and a computer program stored in the memory and executable on the processor, and the processor implements the steps of the target positioning method for sequence data when executing the program.

[0215] In addition, a computer-readable storage medium is provided in an embodiment of the present application, on which a computer program is stored, and when the program is executed by a processor, the steps of the target positioning method of the sequence data are implemented.

[0216] In summary, the embodiment of the present invention uses deep learning to train the neural network model of each human key point, accurately filters the credibility of each human key point involved in the human body part, and filters out the human key points that may not exist, so as to achieve accurate mapping of various parts of the human body. Compared with the prior art method of applying stickers by setting the threshold of human key points, the accuracy of the mapping is further improved. Figure 5 Schematic diagram of sticker effect according to an embodiment of the present invention.

[0217] The present invention realizes accurate image processing technology for each specific position of the target object. Thereby, the target object image processing technology is made more accurate and perfect. Through image processing methods such as stickers, blurring, mosaics, enlargement, reduction, flipping a certain angle, mirror processing, deformation, etc., the specific position of the target object can be accurately covered, highlighted, or other experience effects can be achieved. The above description of accurate image processing takes human body stickers as an example. Other image processing methods, such as enlarging the image, when determining that the specific position of the target object is to be processed by the image, the specific position of the target object is accurately enlarged in proportion to achieve the purpose of the present invention; for another example, deforming the image, when determining that the specific position of the target object is to be processed by the image, the specific position of the target object is deformed to a predetermined degree to achieve the purpose of the present invention. I will not go into details here.

[0218] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for performing image processing on a specific position of a target object, characterized in that: The specific position of the target object involves at least one key point, and the method comprises: Collect pictures containing target objects, train each key point neural network model, and obtain each key point neural network model after training to determine whether the corresponding key point is contained in the picture; Inputting each key point information related to a specific position of a target object in a target image into a corresponding neural network model to determine whether each key point is in the target image; When all the key points involved are included in the target image, determining to perform image processing on a specific position of the target object; There are multiple key points, each key point is associated with two other adjacent key points, the acquisition includes a picture of the target object, and each key point neural network model is trained to obtain each key point neural network model after training. When the key point is the first key point, specifically includes: M pictures are collected, and the credibility of the first key point in each picture, the credibility of the two key points associated with the first key point, and the angle between the first key point and the two associated key points are used as the i-th sample in the deep learning training set, i∈M, M is a natural number; each collected picture includes all specific positions of the target object, or part of the specific positions of the target object; Input the information of the i-th sample into the first key point neural network model to obtain a prediction value of whether the first key point is included in the image; According to the predicted value and the true value of whether the i-th sample is contained in the picture, a loss function is constructed, the loss function is minimized for optimization, the network weight parameters are updated, and the first key point neural network model after training is obtained.

2. The method according to claim 1, characterized in that The step of inputting each key point information involved in the specific position of the target object in the target image into the corresponding neural network model to respectively determine whether each key point is in the target image, when the key points involved include the first key point, specifically includes: The credibility of the first key point, the credibility of the two key points associated with the first key point, and the angle between the first key point and the two associated key points are input into the trained first key point neural network model to obtain whether the first key point is included in the target image.

3. The method according to claim 1, characterized in that The image processing includes performing at least one of the following operations on a specific position of the target object: pasting, blurring, mosaicing, zooming in, zooming out, flipping a certain angle, mirroring, and deformation.

4. A device for performing image processing on a specific position of a target object, characterized in that: The specific position of the target object involves at least one key point, and the device comprises: The training module collects pictures containing the target object, trains the neural network model of each key point, and obtains the trained neural network model of each key point to determine whether the corresponding key point is included in the picture; A processing module inputs each key point information related to a specific position of a target object in a target image into a corresponding neural network model to determine whether each key point is in the target image; A judgment module, when all the key points involved are included in the target image, determines to perform image processing on a specific position of the target object; There are multiple key points, each of which is associated with two other adjacent key points. The training module collects pictures containing the target object, trains each key point neural network model, and obtains each key point neural network model after training. When the key point is the first key point, it is specifically used for: M pictures are collected, and the credibility of the first key point in each picture, the credibility of the two key points associated with the first key point, and the angle between the first key point and the two associated key points are used as the i-th sample in the deep learning training set, i∈M, M is a natural number; each collected picture includes all specific positions of the target object, or part of the specific positions of the target object; Input the information of the i-th sample into the first key point neural network model to obtain a prediction value of whether the first key point is included in the image; According to the predicted value and the true value of whether the i-th sample is contained in the picture, a loss function is constructed, the loss function is minimized for optimization, the network weight parameters are updated, and the first key point neural network model after training is obtained.

5. The device according to claim 4, characterized in that The processing module inputs each key point information involved in the specific position of the target object in the target image into the corresponding neural network model to respectively determine whether each key point is in the target image, and when the key points involved include the first key point, specifically for: The credibility of the first key point, the credibility of the two key points associated with the first key point, and the angle between the first key point and the two associated key points are input into the trained first key point neural network model to obtain whether the first key point is included in the target image.

6. The device according to claim 4, characterized in that The image processing includes performing at least one of the following operations on a specific position of the target object: pasting, blurring, mosaicing, zooming in, zooming out, flipping a certain angle, mirroring, and deformation.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 3 is implemented.

8. A computer readable medium having a computer program stored thereon, wherein: When the program is executed by a processor, the method according to any one of claims 1 to 3 is implemented.

Images