A makeup progress detection method, device, equipment and storage medium
By using facial landmark detection and image processing technology, the eyebrow area is extracted from makeup videos. HSV color space conversion and difference calculation are used to solve the problems of real-time performance and accuracy in makeup progress detection, achieving low-cost and high-efficiency makeup progress detection.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SO-YOUNG INT INC
- Filing Date
- 2021-08-31
- Publication Date
- 2026-07-07
AI Technical Summary
Current technologies for makeup progress detection require high real-time performance, but the accuracy of deep learning models is not high, and a large number of facial images need to be collected, which is costly and makes it difficult to meet the requirements of real-time performance and accuracy.
By acquiring the initial and current frames of a user's makeup video, and utilizing facial landmark detection and image processing techniques, the eyebrow area is extracted from the image, and HSV color space conversion and difference calculation are performed to determine the makeup progress, without the need for a deep learning model.
It achieves highly accurate makeup progress detection, reduces computational costs, lowers server processing pressure, and meets real-time requirements.
Smart Images
Figure CN115761827B_ABST
Abstract
Description
Technical Field
[0001] This application belongs to the field of image processing technology, specifically relating to a method, apparatus, device, and storage medium for detecting makeup progress. Background Technology
[0002] Makeup has become an essential part of many people's daily lives. The makeup process involves many steps. If the makeup progress could be fed back to the user in real time, it would greatly reduce the energy and time spent on makeup.
[0003] Currently, some related technologies use deep learning models to provide functions such as virtual makeup try-on, skin tone detection, and personalized product recommendations. These functions all require the prior collection of a large number of facial images to train the deep learning model.
[0004] However, facial images are private user data, making it difficult to collect a large number of facial images. Furthermore, model training requires significant computational resources, resulting in high costs. The accuracy of a model is inversely proportional to its real-time performance; makeup progress detection requires real-time capture of the user's facial information to determine their current makeup progress, demanding high real-time performance. Deep learning models that can meet these real-time requirements often have low detection accuracy. Summary of the Invention
[0005] This application proposes a method, apparatus, device, and storage medium for detecting makeup progress. It extracts target region images corresponding to the eyebrows from an initial frame image and a current frame image, respectively, and determines the eyebrow makeup progress based on the extracted target region images. This method does not employ deep learning and does not require the pre-collection of large amounts of data. By capturing real-time footage of the user applying makeup, the server calculates the results and returns them to the user, thus reducing computational costs and server processing load.
[0006] The first aspect of this application provides a method for detecting makeup progress, including:
[0007] Obtain the initial frame image and the current frame image from the real-time makeup video of the user performing a specific makeup look;
[0008] Obtain a first target region image corresponding to the eyebrow from the initial frame image, and obtain a second target region image corresponding to the eyebrow from the current frame image;
[0009] Based on the first target region image and the second target region image, determine the current makeup progress corresponding to the current frame image.
[0010] In some embodiments of this application, obtaining the first target region image corresponding to the eyebrow from the initial frame image includes:
[0011] Detect the first facial key points corresponding to the initial frame image;
[0012] Based on the first facial key points, obtain the facial region image corresponding to the initial frame image;
[0013] Based on the eyebrow key points included in the first facial key points, the first target region image corresponding to the eyebrow is obtained from the facial region image.
[0014] In some embodiments of this application, the step of extracting a first target region image corresponding to the eyebrows from the face region image based on the eyebrow key points included in the first facial key points includes:
[0015] Interpolate the eyebrow key points between the inner corner and the peak of the eyebrow, which are included in the first facial key points, to obtain multiple interpolation points;
[0016] Extract all key eyebrow points from the inner corner of the eyebrow to the brow peak and the closed region formed by connecting the multiple interpolation points from the face region image to obtain a partial eyebrow image from the inner corner of the eyebrow to the brow peak.
[0017] A closed region formed by connecting all key eyebrow points from the brow peak to the brow tail is extracted from the face region image to obtain a partial eyebrow image from the brow peak to the brow tail.
[0018] The image of the eyebrow between the inner corner and the arch is stitched together with the image of the eyebrow between the arch and the tail to form the first target region image corresponding to the eyebrow.
[0019] In some embodiments of this application, determining the current makeup progress corresponding to the current frame image based on the first target region image and the second target region image includes:
[0020] The first target region image and the second target region image are respectively converted into images containing preset single-channel components in the HSV color space;
[0021] Based on the converted first target region image and second target region image, the current makeup progress corresponding to the current frame image is determined.
[0022] In some embodiments of this application, determining the current makeup progress corresponding to the current frame image based on the converted first target region image and the second target region image includes:
[0023] Calculate the absolute value of the difference between the preset single-channel components corresponding to pixels with the same position in the converted first target region image and second target region image respectively;
[0024] Count the number of pixels whose absolute values of the corresponding differences satisfy the preset makeup completion conditions;
[0025] The current makeup progress corresponding to the current frame image is obtained by calculating the ratio between the statistically calculated number of pixels and the total number of pixels in all target makeup areas in the first target area image.
[0026] In some embodiments of this application, before determining the current makeup progress corresponding to the current frame image based on the first target region image and the second target region image, the method further includes:
[0027] The first target region image and the second target region image are binarized respectively to obtain a first binarized mask image corresponding to the first target region image and a second binarized mask image corresponding to the second target region image;
[0028] Perform an AND operation on the first binarized mask image and the second binarized mask image to obtain the second mask image corresponding to the intersection region of the first target region image and the second target region image;
[0029] Obtain the face region image corresponding to the initial frame image and the face region image corresponding to the current frame image;
[0030] Perform a bitwise AND operation between the second mask image and the face region image corresponding to the initial frame image to obtain a new first target region image corresponding to the initial frame image;
[0031] Perform a bitwise AND operation between the second mask image and the face region image corresponding to the current frame image to obtain a new second target region image corresponding to the current frame image.
[0032] In some embodiments of this application, before determining the current makeup progress corresponding to the current frame image, the method further includes:
[0033] Boundary erosion processing is performed on the makeup areas in the first target region image and the second target region image, respectively.
[0034] In some embodiments of this application, obtaining the face region image corresponding to the initial frame image based on the first facial key points includes:
[0035] Based on the first facial key points corresponding to the initial frame image, the initial frame image and the first facial key points are rotated and corrected.
[0036] Based on the corrected first facial key points, an image containing the facial region is extracted from the corrected initial frame image;
[0037] The image containing the face region is scaled to a preset size to obtain the face region image corresponding to the initial frame image.
[0038] In some embodiments of this application, the step of rotating and correcting the initial frame image and the first facial key points based on the first facial key points includes:
[0039] Based on the left eye key point and the right eye key point included in the first facial key point, the coordinates of the left eye center and the right eye center are determined respectively;
[0040] Based on the coordinates of the left eye center and the right eye center, determine the rotation angle and the coordinates of the rotation center point corresponding to the initial frame image;
[0041] Based on the rotation angle and the coordinates of the rotation center point, the initial frame image and the first facial key points are rotated and corrected.
[0042] In some embodiments of this application, the step of extracting an image containing a face region from the corrected initial frame image based on the corrected first facial landmarks includes:
[0043] Based on the corrected first facial key points, the facial region contained in the corrected initial frame image is cropped.
[0044] In some embodiments of this application, the step of cropping the face region contained in the corrected initial frame image based on the corrected first facial key points includes:
[0045] Determine the minimum x-coordinate value, minimum y-coordinate value, maximum x-coordinate value, and maximum y-coordinate value from the corrected first facial key points;
[0046] Based on the minimum horizontal coordinate value, the minimum vertical coordinate value, the maximum horizontal coordinate value, and the maximum vertical coordinate value, determine the cropping box corresponding to the face region in the corrected initial frame image;
[0047] Based on the cropping box, an image containing the face region is cropped from the corrected initial frame image.
[0048] In some embodiments of this application, the method further includes:
[0049] Enlarge the cropping frame by a preset factor;
[0050] Based on the magnified cropping frame, an image containing the face region is cropped from the corrected initial frame image.
[0051] In some embodiments of this application, the method further includes:
[0052] Based on the size of the image containing the face region and the preset size, the corrected first facial key points are scaled and translated.
[0053] In some embodiments of this application, the method further includes:
[0054] Detect whether the initial frame image and the current frame image both contain only the face image of the same user;
[0055] If so, then perform the operation of determining the current makeup progress of the user for the specific makeup look;
[0056] If not, a prompt message is sent to the user's terminal, the prompt message being used to remind the user to ensure that only the same user's face appears in the real-time makeup video.
[0057] An embodiment of the second aspect of this application provides a makeup progress detection device, comprising:
[0058] The video acquisition module is used to acquire the initial frame image and the current frame image from the real-time makeup video of the user performing a specific makeup look;
[0059] The target region acquisition module is used to acquire a first target region image corresponding to the eyebrow from the initial frame image, and to acquire a second target region image corresponding to the eyebrow from the current frame image;
[0060] The progress determination module is used to determine the current makeup progress corresponding to the current frame image based on the first target region image and the second target region image.
[0061] An embodiment of the third aspect of this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described in the first aspect above.
[0062] An embodiment of the fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, the program being executed by a processor to implement the method described in the first aspect above.
[0063] The technical solutions provided in this application embodiment have at least the following technical effects or advantages:
[0064] In this embodiment, facial key points are used to correct and crop the user's face region in video frames, improving the accuracy of face region recognition. Based on facial key points, the target region image corresponding to the eyebrows is extracted from the face region image, and pixel alignment is performed on the target region images corresponding to the initial frame image and the current frame image, improving the accuracy of the target region image corresponding to the eyebrows. Alignment of the target makeup region in the initial frame image and the current frame image reduces errors introduced by positional differences in the target makeup region. A piecewise interpolation algorithm is introduced when extracting the eyebrow region, making the extracted eyebrow region more coherent and accurate. Furthermore, this application does not employ deep learning, eliminating the need for pre-collecting large amounts of data. This application captures real-time footage of the user applying makeup, performs server-side calculations, and returns the detection results to the user. Compared to deep learning model inference schemes, this application consumes less computational cost in the algorithm processing stage, reducing the processing pressure on the server.
[0065] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description
[0066] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings.
[0067] In the attached diagram:
[0068] Figure 1 A flowchart of a makeup progress detection method for detecting eyebrow makeup provided in an embodiment of this application is shown;
[0069] Figure 2 A schematic diagram showing the rotation angle of the solved image provided in an embodiment of this application is illustrated;
[0070] Figure 3 A schematic diagram illustrating two coordinate system transformations provided in an embodiment of this application is shown;
[0071] Figure 4 A schematic diagram of the module flow of a makeup progress detection method for detecting eyebrow makeup provided in an embodiment of this application is shown.
[0072] Figure 5 This invention provides a schematic diagram of the structure of a makeup progress detection device for detecting eyebrow makeup according to an embodiment of the present application.
[0073] Figure 6This illustration shows a schematic diagram of the structure of an electronic device according to an embodiment of this application;
[0074] Figure 7 A schematic diagram of a storage medium provided in one embodiment of this application is shown. Detailed Implementation
[0075] Exemplary embodiments of this application will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of this application are shown in the drawings, it should be understood that this application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of this application and to fully convey the scope of this application to those skilled in the art.
[0076] It should be noted that, unless otherwise stated, the technical or scientific terms used in this application shall have the ordinary meaning as understood by one of ordinary skill in the art to which this application pertains.
[0077] The following description, in conjunction with the accompanying drawings, describes a method, apparatus, device, and storage medium for detecting makeup progress according to embodiments of this application.
[0078] Currently, some related technologies offer virtual makeup try-on functions, which can be applied at sales counters or mobile applications. These technologies use facial recognition to provide users with virtual makeup try-on services, allowing for the matching of various makeup looks and real-time facial fit. Others offer facial skin detection services, but these services only address users' needs for selecting suitable cosmetics or skincare routines. While these services can help users choose suitable eyebrow makeup products, they cannot display the makeup application progress and cannot meet users' real-time makeup application needs. Related technologies also use deep learning models to provide virtual makeup try-on, skin tone detection, and personalized product recommendations. These functions all require the pre-collection of a large number of facial images to train the deep learning model. However, facial images are users' private data, making it difficult to collect a large number of facial images. Furthermore, model training consumes significant computational resources, resulting in high costs. The accuracy of the model is inversely proportional to its real-time performance. Makeup progress detection requires real-time capture of the user's facial information to determine the current makeup progress, which has very high real-time requirements. Deep learning models that can meet real-time requirements often have low detection accuracy.
[0079] Based on this, this application provides a makeup progress detection method. This method can detect the eyebrow makeup progress solely through image processing, achieving high accuracy. It can detect the user's makeup progress in real time during the eyebrow makeup process. It eliminates the need for deep learning models, resulting in low computational load and cost, reducing server processing pressure, improving the efficiency of makeup progress detection, and meeting the real-time requirements of makeup progress detection.
[0080] See Figure 1 This method is used to describe the makeup progress corresponding to eyebrow makeup, and it specifically includes the following steps:
[0081] Step 601: Obtain the initial frame image and the current frame image from the real-time makeup video of the user performing a specific makeup look.
[0082] The execution entity in this embodiment is a server. A client compatible with the makeup progress detection service provided by the server is installed on the user's mobile phone or computer. When the user needs to use the makeup progress detection service, the user opens the client on their terminal. The client's interface has a video upload interface. When the system detects that the user clicks on the video upload interface, it calls the terminal's camera to record a makeup video of the user. During the recording, the user applies makeup to their eyebrows. The user's terminal transmits the recorded makeup video to the server as a video stream. The server receives each frame of the makeup video transmitted by the user's terminal.
[0083] In this embodiment, the server uses the first received frame image as the initial frame image and uses this initial frame image as a reference to compare the current makeup progress of the specific makeup look corresponding to each subsequent received frame image. Since the processing method for each subsequent frame image is the same, this embodiment uses the current frame image received at the current moment as an example to illustrate the makeup progress detection process.
[0084] In other embodiments of this application, after obtaining the initial frame image and the current frame image of the user's makeup video, the server further detects whether both the initial frame image and the current frame image contain only the face image of the same user. First, it detects whether both the initial frame image and the current frame image contain only one face image. If the initial frame image and / or the current frame image contain multiple face images, or if the initial frame image and / or the current frame image do not contain any face images, a prompt message is sent to the user's terminal. The user's terminal receives and displays the prompt message to remind the user to ensure that only the face of the same user appears in the makeup video. For example, the prompt message could be "Please ensure that only the face of the same person appears in the shot."
[0085] If both the initial frame and the current frame contain only one face image, the system further determines whether the face images in the initial frame and the current frame belong to the same user. Specifically, facial feature information corresponding to the face image in the initial frame and the face image in the current frame can be extracted using facial recognition technology. The similarity of the extracted facial feature information from these two frames is calculated. If the calculated similarity is greater than or equal to a set value, it is determined that the faces in the initial frame and the current frame belong to the same user. If the calculated similarity is less than the set value, it is determined that the faces in the initial frame and the current frame belong to different users, and a prompt message is sent to the user's terminal. The user's terminal receives and displays this prompt message to remind the user to ensure that only the face of the same user appears in the makeup video.
[0086] After obtaining the initial frame image and the current frame image of the user's makeup process in this step, the server determines the user's current makeup progress through the operations in steps 602 and 603 below.
[0087] Step 602: Obtain the first target region image corresponding to the eyebrow from the initial frame image, and obtain the second target region image corresponding to the eyebrow from the current frame image.
[0088] The process of acquiring the first target region image is the same as the process of acquiring the second target region image. This embodiment uses the acquisition process of the first target region image as an example for detailed explanation. Specifically, the server obtains the first target region image from the initial frame image through the following steps S5-S7.
[0089] S5: Detect the first facial landmarks corresponding to the initial frame image.
[0090] The server is configured with a pre-trained detection model for detecting facial landmarks, and provides an interface service for facial landmark detection through this model. After the server obtains the initial frame image of the user's makeup video, it calls the facial landmark detection interface service and uses the detection model to identify all facial landmarks on the user's face in the initial frame image. To distinguish them from the facial landmarks corresponding to the current frame image, this embodiment refers to all facial landmarks corresponding to the initial frame image as the first facial landmarks. All facial landmarks corresponding to the current frame image are referred to as the second facial landmarks.
[0091] The identified facial landmarks include key points on the user's facial contours, as well as key points on the mouth, nose, eyes, eyebrows, and other parts of the face. The number of identified facial landmarks can be up to 106.
[0092] S6: Based on the first facial key points, obtain the facial region image corresponding to the initial frame image.
[0093] The server obtains the face region image corresponding to the initial frame image through the following steps S60-S62:
[0094] S60: Based on the first facial key points, perform rotation correction on the initial frame image and the first facial key points.
[0095] Since users cannot guarantee that the facial pose angles are the same in every frame when shooting makeup videos through the terminal, in order to improve the accuracy of comparing the current frame image with the initial frame image, it is necessary to rotate and correct the face in each frame image so that the line connecting the eyes of the face in each frame image is on the same horizontal line after correction. This ensures that the facial pose angles are the same in each frame image and avoids the problem of large errors in makeup progress detection due to different pose angles.
[0096] Specifically, based on the left and right eye keypoints included in the first facial keypoint system, the center coordinates of the left and right eyes are determined respectively. All left-eye keypoints in the left eye region and all right-eye keypoints in the right eye region are determined from the first facial keypoint system. The average of the x-coordinates and y-coordinates of all determined left-eye keypoints is taken, and the average of these two values is combined to form a coordinate system, which is then used as the center coordinate of the left eye. The center coordinates of the right eye are determined in the same way.
[0097] Then, based on the coordinates of the left and right eye centers, the rotation angle and rotation center coordinates corresponding to the initial frame image are determined. For example... Figure 2 As shown, the horizontal difference (dx) and vertical difference (dy) between the left and right eye center coordinates are calculated, along with the length (d) of the line connecting the left and right eye center coordinates. Based on the length (d), horizontal difference (dx), and vertical difference (dy), the angle (θ) between the line and the horizontal direction is calculated; this angle (θ) is the rotation angle corresponding to the initial frame image. Then, the coordinates of the center point of the line connecting the left and right eye center coordinates are calculated; this midpoint coordinate is the rotation center point coordinate corresponding to the initial frame image.
[0098] Based on the calculated rotation angle and rotation center point coordinates, the initial frame image and the first facial key points are rotated and corrected. Specifically, the rotation angle and rotation center point coordinates are input into a preset function for calculating the image's rotation matrix. This preset function can be the OpenCV function `cv2.getRotationMatrix2D()`. The rotation matrix corresponding to the initial frame image is obtained by calling this preset function. Then, the product of the initial frame image and the rotation matrix is calculated to obtain the corrected initial frame image. The operation of correcting the initial frame image using the rotation matrix can also be accomplished by calling the OpenCV function `cv2.warpAffine()`.
[0099] For the first facial key points, each key point needs to be corrected individually to correspond with the corrected initial frame image. During the correction of each key point, two coordinate system transformations are required. The first transformation converts the coordinate system originating from the top left corner of the initial frame image to the bottom left corner. The second transformation further converts the bottom left corner coordinate system to a coordinate system with the aforementioned rotation center point coordinates as the origin. Figure 3 As shown. After two coordinate system transformations, each first facial key point is transformed using the following formula (1) to complete the rotation correction of the first facial key points.
[0100]
[0101] In formula (1), x0 and y0 are the abscissa and ordinate of the first facial key point before rotation correction, respectively, x and y are the abscissa and ordinate of the first facial key point after rotation correction, and θ is the above rotation angle.
[0102] The corrected initial frame image and the first facial key points are based on the entire image. The entire image not only contains the user's facial information, but also other redundant image information. Therefore, the face region needs to be cropped in the corrected image through the following step S61.
[0103] S61: Based on the corrected first facial landmarks, extract an image containing the facial region from the corrected initial frame image.
[0104] First, determine the minimum, minimum, maximum, and maximum x and y coordinates from the corrected first facial landmarks. Then, based on these values, determine the cropping box corresponding to the face region in the corrected initial frame image. Specifically, combine the minimum and minimum x and y coordinates to form a coordinate point, which is used as the top-left vertex of the cropping box corresponding to the face region. Combine the maximum and maximum x and y coordinates to form another coordinate point, which is used as the bottom-right vertex of the cropping box corresponding to the face region. Based on these top-left and bottom-right vertices in the corrected initial frame image, determine the position of the cropping box, and then extract the image within this cropping box from the corrected initial frame image, i.e., extract the image containing the face region.
[0105] In other embodiments of this application, to ensure that the entire facial region of the user is captured and to avoid significant errors in subsequent makeup progress detection due to incomplete capture, the capture frame can be enlarged by a preset factor, such as 1.15 or 1.25. This application does not limit the specific value of the preset factor; it can be set according to requirements in practical applications. After enlarging the capture frame by the preset factor, the image located within the enlarged capture frame is extracted from the corrected initial frame image, thereby capturing an image containing the complete facial region of the user.
[0106] S62: Scale the image containing the face region to a preset size to obtain the face region image corresponding to the initial frame image.
[0107] After extracting the image containing the user's face region from the initial frame image using the above method, the image containing the face region is scaled to a preset size to obtain the face region image corresponding to the initial frame image. This preset size can be 390×390 or 400×400, etc. This application embodiment does not limit the specific value of the preset size; it can be set according to requirements in practical applications.
[0108] To adapt the first facial landmark to the scaled face region image, after scaling the cropped image containing the face region to a preset size, the corrected first facial landmark needs to be scaled and translated based on the size of the image containing the face region before scaling and the preset size. Specifically, based on the size of the image containing the face region before scaling and the preset size to which the image needs to be scaled, the translation direction and translation distance of each first facial landmark are determined. Then, according to the translation direction and translation distance corresponding to each first facial landmark, the translation operation is performed on each first facial landmark, and the coordinates of each first facial landmark after translation are recorded.
[0109] The face region image is obtained from the initial frame image in the above manner, and the first face key points are adapted to the obtained face region image through operations such as rotation correction and translation scaling. Then, the first target region image corresponding to the eyebrows is extracted from the face region image in the following step S7.
[0110] In some other embodiments of this application, before performing step S7, the face region image may be subjected to Gaussian filtering to remove noise from the face region image. Specifically, Gaussian filtering is performed on the face region image corresponding to the initial frame image according to a Gaussian kernel of a preset size.
[0111] The Gaussian kernel is a key parameter in Gaussian filtering. If the kernel is too small, the filtering effect will be poor; if it is too large, while it can filter out noise, it will smooth out useful information. In this embodiment, a Gaussian kernel of a preset size, such as 9×9, is selected. Furthermore, the other two parameters of the Gaussian filtering function, sigmaX and sigmaY, are both set to 0. After Gaussian filtering, the image information is smoother, thereby improving the accuracy of subsequent makeup progress detection.
[0112] The face region image is obtained in the above manner, or after Gaussian filtering the face region image, the first target region image corresponding to the eyebrows is extracted from the face region image corresponding to the initial frame image in step S7.
[0113] S7: Based on the eyebrow key points included in the first face key points, extract the first target region image corresponding to the eyebrow from the face region image corresponding to the initial frame image.
[0114] When performing progress detection for eyebrow makeup, it is necessary to extract the image of the area where the eyebrows are located to avoid the influence of other areas on the progress detection. Moreover, by extracting the eyebrow area, subsequent calculations are only performed on the eyebrow area, which reduces the amount of computation and improves accuracy.
[0115] The first facial key points obtained above include multiple eyebrow key points, such as 18 eyebrow key points. These multiple eyebrow key points are distributed at different positions on the eyebrow contour from the inner corner to the outer corner. To improve the accuracy of extracting the first target region image corresponding to the eyebrow, this embodiment uses linear interpolation to obtain more points on the eyebrow contour, thereby extracting the image based on more points. Since the eyebrow tail is pointed, it is not convenient to perform linear interpolation calculations. Therefore, this embodiment divides the process of extracting the first target region image corresponding to the eyebrow into two parts. One part is the segment from the inner corner to the brow peak, where more points are obtained through linear interpolation to extract the image. The other part is the segment from the brow peak to the outer corner, where the eyebrow key points obtained from the brow peak to the outer corner are used to extract the image.
[0116] Specifically, linear interpolation is performed on the eyebrow key points between the inner corner and the arch of the eyebrow in the first facial key point, resulting in multiple interpolation points. All eyebrow key points between the inner corner and the arch, along with the multiple interpolation points, in the facial region image corresponding to the initial frame image are sequentially connected along the eyebrow contour line to obtain a closed region. This closed region encloses a portion of the eyebrow area from the inner corner to the arch. The image of this closed region is extracted from the facial region image corresponding to the initial frame image to obtain the partial eyebrow image between the inner corner and the arch.
[0117] Connect all eyebrow key points from the brow peak to the brow tail in the face region image corresponding to the initial frame image along the eyebrow outline to obtain a closed region. This closed region encloses a portion of the eyebrow area from the brow peak to the brow tail. Extract this closed region from the face region image corresponding to the initial frame image to obtain the partial eyebrow image from the brow peak to the brow tail.
[0118] The image of the eyebrow from the inner corner to the arch is stitched together with the image of the eyebrow from the arch to the tail to form the first target region image corresponding to the eyebrow.
[0119] For the current frame image, the same steps S5-S7 above are used to obtain the second target region image corresponding to the eyebrows from the current frame image.
[0120] In other embodiments of this application, considering that the edges of eyebrows applied in actual makeup scenarios may not have very clear outlines and the boundaries are usually blurred to avoid appearing abrupt, after obtaining the first target region image and the second target region image through the above embodiments, boundary erosion processing is performed on the eyebrow region in the first target region image and the second target region image respectively to blur the boundaries of the target makeup region for eyebrows, thereby improving the accuracy of makeup progress detection.
[0121] Step 603: Determine the current makeup progress corresponding to the current frame image based on the first target region image and the second target region image.
[0122] The color space of the first target region image corresponding to the eyebrows in the initial frame image and the second target region image corresponding to the eyebrows in the current frame image obtained by the above method are both RGB color space. This application embodiment pre-determines the impact of eyebrow makeup on each channel component of the color space through numerous experiments, finding that the impact on each color channel in the RGB color space is not significantly different. The HSV color space, however, consists of three components: Hue, Saturation, and Value. When one component changes, the values of the other two components do not change significantly. Compared to the RGB color space, the HSV color space can separate one channel component. Furthermore, experiments determined which channel component among Value, Hue, and Saturation has the greatest impact on eyebrow makeup, and the channel component with the greatest impact is configured in the server as the preset single-channel component corresponding to the preset type of makeup. For eyebrow makeup, the corresponding preset single-channel component can be the Value component.
[0123] After obtaining the first target region image corresponding to the eyebrows in the initial frame image and the second target region image corresponding to the eyebrows in the current frame image using the above method, both the first and second target region images are converted from RGB color space to HSV color space. A preset single-channel component is then extracted from the HSV color space of the converted first target region image to obtain a first target region image containing only the preset single-channel component. Similarly, a preset single-channel component is extracted from the HSV color space of the converted second target region image to obtain a second target region image containing only the preset single-channel component.
[0124] Then, based on the converted first target region image and second target region image, the current makeup progress corresponding to the current frame image is determined.
[0125] Specifically, the absolute values of the differences between the aforementioned channel components corresponding to pixels at the same position in the transformed first and second target region images are calculated. For example, the absolute values of the differences between the luminance components between pixels at the same coordinates in the transformed first and second target region images are calculated. The number of pixels whose corresponding absolute values of difference satisfy the preset makeup completion condition is counted. The preset makeup completion condition is that the absolute value of the difference between the pixels is greater than a first preset threshold, which can be 7 or 8, etc.
[0126] The total number of pixels in the eyebrow region of either the first or second target region image is counted. Then, the ratio between the number of pixels that meet the preset makeup completion conditions and the total number of pixels in the eyebrow region is calculated, and this ratio is determined as the current makeup progress.
[0127] In other embodiments of this application, to further improve the accuracy of makeup progress detection, the eyebrow regions in the first and second target region images are further aligned. Specifically, the first and second target region images, which contain only the aforementioned preset single-channel components, are binarized. Specifically, the values of the preset single-channel components corresponding to pixels in the target makeup region of both the first and second target region images are modified to 1, while the values of the preset single-channel components for pixels at other locations are modified to 0. The binarization process yields a first binarized mask image corresponding to the first target region image and a second binarized mask image corresponding to the second target region image.
[0128] Then, an AND operation is performed on the first and second binarized mask images. Specifically, the pixels at the same positions in both images are ANDed to obtain the second mask image corresponding to the intersection region of the first and second target region images. The regions in this second mask image where the preset single-channel component of a pixel is non-zero represent the overlapping target makeup area in both the first and second target region images.
[0129] The aforementioned steps are used to obtain the face region images corresponding to the initial frame image and the current frame image. A bitwise AND operation is performed between the second mask image and the face region image corresponding to the initial frame image to obtain the new first target region image corresponding to the eyebrows in the initial frame image; a bitwise AND operation is then performed between the second mask image and the face region image corresponding to the current frame image to obtain the new second target region image corresponding to the eyebrows in the current frame image.
[0130] In other embodiments of this application, a bitwise AND operation can be performed on the second mask image and the first target region image corresponding to the eyebrow after boundary erosion processing to obtain a new first target region image corresponding to the eyebrow. Similarly, a bitwise AND operation can be performed on the second mask image and the second target region image corresponding to the eyebrow after boundary erosion processing to obtain a new second target region image corresponding to the eyebrow.
[0131] Since the second mask image contains the target makeup area that overlaps in the initial frame image and the current frame image, the new first target area image and the new second target area image are extracted from the second mask image in the manner described above. This ensures that the position of the target makeup area in the new first target area image and the new second target area image is completely consistent. In this way, the makeup progress is determined by comparing the changes in the target makeup area in the current frame image with those in the initial frame image. This ensures that the areas being compared are completely consistent, greatly improving the accuracy of makeup progress detection.
[0132] After aligning the target makeup areas in the initial frame image and the current frame image in the above manner to obtain the new first target area image and the new second target area image, the current makeup progress corresponding to the current frame image is determined again through the operation of step 603 above.
[0133] After determining the current makeup progress using any of the methods described above, the server sends this progress information to the user's terminal. Upon receiving the progress information, the user's terminal displays it. The current makeup progress can be a ratio or a percentage. The terminal can display the current makeup progress as a progress bar.
[0134] During the user's makeup application process, the makeup progress detection method provided in this application embodiment detects the makeup progress of each subsequent frame relative to the first frame in real time, and displays the detected makeup progress to the user, allowing the user to intuitively see their makeup progress and improve makeup efficiency.
[0135] To facilitate understanding of the methods provided in the embodiments of this application, the following description is provided in conjunction with the accompanying drawings. Figure 4 As shown, based on the initial frame image and its corresponding first facial key points, and the current frame image and its corresponding second facial key points, the faces in the initial frame image and the current frame image are aligned and cropped respectively. Then, the Laplacian algorithm is used to smooth and denoise the two cropped face region images. Next, the first target region image and the second target region image corresponding to the eyebrows are extracted from the two face region images respectively. Boundary erosion processing is performed on the first target region image and the second target region image. Then, the first target region image and the second target region image are converted into images containing only a preset single-channel component in the HSV color space. The first target region image and the second target region image are aligned again, and then the current makeup progress is calculated based on both.
[0136] In this embodiment, facial key points are used to correct and crop the user's face region in video frames, improving the accuracy of face region recognition. Based on facial key points, the target region image corresponding to the eyebrows is extracted from the face region image, and pixel alignment is performed on the target region images corresponding to the initial frame image and the current frame image, improving the accuracy of the target region image corresponding to the eyebrows. Alignment of the target makeup region in the initial frame image and the current frame image reduces errors introduced by positional differences in the target makeup region. A piecewise interpolation algorithm is introduced when extracting the eyebrow region, making the extracted eyebrow region more coherent and accurate. Furthermore, this application does not employ deep learning, eliminating the need for pre-collecting large amounts of data. This application captures real-time footage of the user applying makeup, performs server-side calculations, and returns the detection results to the user. Compared to deep learning model inference schemes, this application consumes less computational cost in the algorithm processing stage, reducing the processing pressure on the server.
[0137] This application also provides a makeup progress detection device for performing the above-described makeup progress detection method for detecting the makeup progress of eyebrows. See also... Figure 5 The device specifically includes:
[0138] The video acquisition module 1601 is used to acquire the initial frame image and the current frame image in a real-time makeup video of the user performing a specific makeup look;
[0139] The target region acquisition module 1602 is used to acquire a first target region image corresponding to the eyebrow from an initial frame image, and to acquire a second target region image corresponding to the eyebrow from a current frame image;
[0140] The progress determination module 1603 is used to determine the current makeup progress corresponding to the current frame image based on the first target area image and the second target area image.
[0141] The target region acquisition module 1602 is used to detect the first facial key points corresponding to the initial frame image; acquire the facial region image corresponding to the initial frame image based on the first facial key points; and acquire the first target region image corresponding to the eyebrows from the facial region image based on the eyebrow key points included in the first facial key points.
[0142] The target region acquisition module 1602 is used to interpolate the eyebrow key points from the inner corner of the eyebrow to the brow peak, including the first facial key points, to obtain multiple interpolation points; to extract a closed region formed by connecting all eyebrow key points from the inner corner of the eyebrow to the brow peak and multiple interpolation points from the facial region image, to obtain a partial eyebrow image from the inner corner of the eyebrow to the brow peak; to extract a closed region formed by connecting all eyebrow key points from the brow peak to the eyebrow tail from the facial region image, to obtain a partial eyebrow image from the brow peak to the eyebrow tail; and to stitch the partial eyebrow image from the inner corner of the eyebrow to the brow peak and the partial eyebrow image from the brow peak to the eyebrow tail together to form the first target region image corresponding to the eyebrow.
[0143] The progress determination module 1603 is used to convert the first target region image and the second target region image into images containing preset single-channel components in the HSV color space, respectively; and to determine the current makeup progress corresponding to the current frame image based on the converted first target region image and the second target region image.
[0144] The progress determination module 1603 is used to calculate the absolute value of the difference between the preset single-channel components corresponding to the pixels with the same position in the converted first target region image and the second target region image respectively; count the number of pixels whose corresponding absolute difference satisfies the preset makeup completion condition; and calculate the ratio between the counted number of pixels and the total number of pixels in all target makeup regions in the first target region image to obtain the current makeup progress corresponding to the current frame image.
[0145] The progress determination module 1603 is further configured to perform binarization processing on the first target region image and the second target region image respectively, to obtain a first binarized mask image corresponding to the first target region image and a second binarized mask image corresponding to the second target region image; perform an AND operation on the first binarized mask image and the second binarized mask image to obtain a second mask image corresponding to the intersection region of the first target region image and the second target region image; acquire the face region image corresponding to the initial frame image and the face region image corresponding to the current frame image; perform an AND operation on the second mask image and the face region image corresponding to the initial frame image to obtain a new first target region image corresponding to the initial frame image; and perform an AND operation on the second mask image and the face region image corresponding to the current frame image to obtain a new second target region image corresponding to the current frame image.
[0146] The device also includes a boundary erosion module, used to perform boundary erosion processing on the makeup areas in the first target area image and the second target area image, respectively.
[0147] The target region acquisition module 1602 is used to rotate and correct the initial frame image and the first facial key points according to the first facial key points corresponding to the initial frame image; to extract an image containing the face region from the corrected initial frame image according to the corrected first facial key points; and to scale the image containing the face region to a preset size to obtain the face region image corresponding to the initial frame image.
[0148] The target region acquisition module 1602 is used to determine the center coordinates of the left eye and the center coordinates of the right eye based on the left eye key points and the right eye key points included in the first facial key points; determine the rotation angle and rotation center point coordinates corresponding to the initial frame image based on the left eye center coordinates and the right eye center coordinates; and perform rotation correction on the initial frame image and the first facial key points based on the rotation angle and rotation center point coordinates.
[0149] The target region acquisition module 1602 is used to extract the face region contained in the corrected initial frame image based on the corrected first face key points.
[0150] The target region acquisition module 1602 is used to determine the minimum horizontal coordinate value, minimum vertical coordinate value, maximum horizontal coordinate value, and maximum vertical coordinate value from the corrected first facial key points; determine the cropping box corresponding to the face region in the corrected initial frame image based on the minimum horizontal coordinate value, minimum vertical coordinate value, maximum horizontal coordinate value, and maximum vertical coordinate value; and extract the image containing the face region from the corrected initial frame image based on the cropping box.
[0151] The target region acquisition module 1602 is also used to enlarge the cropping box by a preset factor; based on the enlarged cropping box, to extract an image containing the face region from the corrected initial frame image.
[0152] The target region acquisition module 1602 is also used to scale and translate the corrected first facial key points according to the size of the image containing the face region and the preset size.
[0153] The device also includes a face detection module, used to detect whether the initial frame image and the current frame image both contain only the face image of the same user; if so, it performs the operation of determining the current makeup progress of the user performing a specific makeup look; if not, it sends a prompt message to the user's terminal, the prompt message being used to remind the user to ensure that only the face of the same user appears in the real-time makeup video.
[0154] The makeup progress detection device and the makeup progress detection method provided in the above embodiments of this application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the applications stored therein.
[0155] This application also provides an electronic device for performing the above-described makeup progress detection method. Please refer to... Figure 6 This illustrates a schematic diagram of an electronic device provided by some embodiments of this application. For example... Figure 6 As shown, the electronic device 11 includes: a processor 1100, a memory 1101, a bus 1102, and a communication interface 1103. The processor 1100, the communication interface 1103, and the memory 1101 are connected via the bus 1102. The memory 1101 stores a computer program that can run on the processor 1100. When the processor 1100 runs the computer program, it executes the makeup progress detection method provided in any of the foregoing embodiments of this application.
[0156] The memory 1101 may include high-speed random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Communication between this device network element and at least one other network element is achieved through at least one communication interface 1103 (which can be wired or wireless), such as the Internet, wide area network, local area network, metropolitan area network, etc.
[0157] Bus 1102 can be an ISA bus, PCI bus, or EISA bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. The memory 1101 is used to store programs. After receiving an execution instruction, the processor 1100 executes the program. The makeup progress detection method disclosed in any of the foregoing embodiments of this application can be applied to the processor 1100, or implemented by the processor 1100.
[0158] The processor 1100 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of the processor 1100 or by instructions in software form. The processor 1100 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules may reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 1101. Processor 1100 reads the information in memory 1101 and, in conjunction with its hardware, completes the steps of the above method.
[0159] The electronic device provided in this application embodiment and the makeup progress detection method provided in this application embodiment are based on the same inventive concept and have the same beneficial effects as the methods they adopt, operate or implement.
[0160] This application also provides a computer-readable storage medium corresponding to the makeup progress detection method provided in the foregoing embodiments. Please refer to... Figure 7 The computer-readable storage medium shown is an optical disc 30, on which a computer program (i.e., a program product) is stored. When the computer program is run by a processor, it executes the makeup progress detection method provided in any of the foregoing embodiments.
[0161] It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical and magnetic storage media, which will not be elaborated here.
[0162] The computer-readable storage medium provided in the above embodiments of this application and the makeup progress detection method provided in the embodiments of this application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the application stored therein.
[0163] It should be noted that:
[0164] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of this application may be practiced without these specific details. In some instances, well-known structures and techniques have not been shown in detail so as not to obscure the understanding of this specification.
[0165] Similarly, it should be understood that, for the sake of brevity and to aid in understanding one or more of the various inventive aspects, in the above description of exemplary embodiments of this application, various features of this application are sometimes grouped together in a single embodiment, figure, or description thereof. However, this disclosure should not be construed as reflecting a schematic diagram in which the claimed application requires more features than expressly recited in each claim. Rather, as reflected in the following claims, inventive aspects lie in fewer than all features of a single foregoing disclosed embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into that detailed description, wherein each claim itself is a separate embodiment of this application.
[0166] Furthermore, those skilled in the art will understand that although some embodiments described herein include certain features but not others included in other embodiments, combinations of features from different embodiments are intended to be within the scope of this application and form different embodiments. For example, in the following claims, any of the claimed embodiments can be used in any combination.
[0167] The above description is merely a preferred embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
Claims
1. A method for detecting makeup progress, characterized in that, include: Obtain the initial frame image and the current frame image from the real-time makeup video of the user performing a specific makeup look; Obtain a first target region image corresponding to the eyebrow from the initial frame image, and obtain a second target region image corresponding to the eyebrow from the current frame image; The first target region image and the second target region image are respectively converted into images containing preset single-channel components in the HSV color space; The first target region image and the second target region image after conversion are binarized respectively to obtain a first binarized mask image corresponding to the first target region image and a second binarized mask image corresponding to the second target region image; The pixels at the same position in the first binarized mask image and the second binarized mask image are ANDed to obtain the second mask image corresponding to the intersection area of the first target region image and the second target region image. The region in the second mask image where the preset single-channel component of a pixel is not zero is the target makeup region that overlaps between the first target region image and the second target region. Perform a bitwise AND operation between the second mask image and the face region image corresponding to the initial frame image to obtain a new first target region image corresponding to the eyebrows in the initial frame image; Perform a bitwise AND operation between the second mask image and the face region image corresponding to the current frame image to obtain a new second target region image corresponding to the eyebrows in the current frame image; Based on the changes in the new first target region image and the new second target region image, the current makeup progress corresponding to the current frame image is determined.
2. The method according to claim 1, characterized in that, The step of obtaining the first target region image corresponding to the eyebrow from the initial frame image includes: Detect the first facial key points corresponding to the initial frame image; Based on the first facial key points, obtain the facial region image corresponding to the initial frame image; Based on the eyebrow key points included in the first facial key points, the first target region image corresponding to the eyebrow is obtained from the facial region image.
3. The method according to claim 2, characterized in that, The step of extracting the first target region image corresponding to the eyebrows from the face region image based on the eyebrow key points included in the first facial key points includes: Interpolate the eyebrow key points between the inner corner and the peak of the eyebrow, which are included in the first facial key points, to obtain multiple interpolation points; Extract all key eyebrow points from the inner corner of the eyebrow to the brow peak and the closed region formed by connecting the multiple interpolation points from the face region image to obtain a partial eyebrow image from the inner corner of the eyebrow to the brow peak. A closed region formed by connecting all key eyebrow points from the brow peak to the brow tail is extracted from the face region image to obtain a partial eyebrow image from the brow peak to the brow tail. The image of the eyebrow between the inner corner and the arch is stitched together with the image of the eyebrow between the arch and the tail to form the first target region image corresponding to the eyebrow.
4. The method according to claim 1, characterized in that, The step of determining the current makeup progress corresponding to the current frame image based on the new first target region image and the new second target region image includes: Calculate the absolute value of the difference between the preset single-channel components corresponding to pixels with the same position in the new first target region image and the new second target region image, respectively; Count the number of pixels whose absolute values of the corresponding differences satisfy the preset makeup completion conditions; The current makeup progress corresponding to the current frame image is obtained by calculating the ratio between the statistically calculated number of pixels and the total number of pixels in all target makeup areas in the new first target area image.
5. The method according to claim 1, characterized in that, Before determining the current makeup progress corresponding to the current frame image, the method further includes: Boundary erosion processing is performed on the makeup areas in the first target region image and the second target region image, respectively.
6. The method according to claim 2, characterized in that, The step of obtaining the face region image corresponding to the initial frame image based on the first facial key points includes: Based on the first facial key points corresponding to the initial frame image, the initial frame image and the first facial key points are rotated and corrected. Based on the corrected first facial key points, an image containing the facial region is extracted from the corrected initial frame image; The image containing the face region is scaled to a preset size to obtain the face region image corresponding to the initial frame image.
7. The method according to claim 6, characterized in that, The step of rotating and correcting the initial frame image and the first facial key points based on the first facial key points includes: Based on the left eye key point and the right eye key point included in the first facial key point, the coordinates of the left eye center and the right eye center are determined respectively; Based on the coordinates of the left eye center and the right eye center, determine the rotation angle and the coordinates of the rotation center point corresponding to the initial frame image; Based on the rotation angle and the coordinates of the rotation center point, the initial frame image and the first facial key points are rotated and corrected.
8. The method according to claim 6, characterized in that, The step of extracting an image containing a face region from the corrected initial frame image based on the corrected first facial key points includes: Based on the corrected first facial key points, the facial region contained in the corrected initial frame image is cropped.
9. The method according to claim 8, characterized in that, The step of cropping the face region contained in the corrected initial frame image based on the corrected first facial key points includes: Determine the minimum x-coordinate value, minimum y-coordinate value, maximum x-coordinate value, and maximum y-coordinate value from the corrected first facial key points; Based on the minimum horizontal coordinate value, the minimum vertical coordinate value, the maximum horizontal coordinate value, and the maximum vertical coordinate value, determine the cropping box corresponding to the face region in the corrected initial frame image; Based on the cropping box, an image containing the face region is cropped from the corrected initial frame image.
10. The method according to claim 9, characterized in that, The method further includes: Enlarge the cropping frame by a preset factor; Based on the magnified cropping frame, an image containing the face region is cropped from the corrected initial frame image.
11. The method according to claim 6, characterized in that, The method further includes: Based on the size of the image containing the face region and the preset size, the corrected first facial key points are scaled and translated.
12. The method according to any one of claims 1-11, characterized in that, The method further includes: Detect whether the initial frame image and the current frame image both contain only the face image of the same user; If so, then perform the operation of determining the current makeup progress of the user for the specific makeup look; If not, a prompt message is sent to the user's terminal, the prompt message being used to remind the user to ensure that only the same user's face appears in the real-time makeup video.
13. A makeup progress detection device, characterized in that, include: The video acquisition module is used to acquire the initial frame image and the current frame image from the real-time makeup video of the user performing a specific makeup look; The target region acquisition module is used to acquire a first target region image corresponding to the eyebrow from the initial frame image, and to acquire a second target region image corresponding to the eyebrow from the current frame image; The progress determination module is used to convert the first target region image and the second target region image into images containing preset single-channel components in the HSV color space, respectively; and to perform binarization processing on the converted first target region image and the second target region image to obtain a first binarized mask image corresponding to the first target region image and a second binarized mask image corresponding to the second target region image. The pixels at the same position in the first binarized mask image and the second binarized mask image are ANDed to obtain the second mask image corresponding to the intersection area of the first target region image and the second target region image. The region in the second mask image where the preset single-channel component of a pixel is not zero is the target makeup region that overlaps between the first target region image and the second target region. Perform an AND operation on the face region image corresponding to the second mask image and the initial frame image to obtain a new first target region image corresponding to the eyebrows in the initial frame image; perform an AND operation on the face region image corresponding to the second mask image and the current frame image to obtain a new second target region image corresponding to the eyebrows in the current frame image; determine the current makeup progress corresponding to the current frame image based on the changes in the new first target region image and the new second target region image.
14. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the method as described in any one of claims 1-12.
15. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by a processor to implement the method as described in any one of claims 1-12.