Intelligent mirror and makeup teaching method

Abstract

The present application provides a smart mirror and a makeup teaching method. The smart mirror includes a display screen for displaying images or videos; a camera for collecting a user's face image; and a processor configured to: respond to a user's instruction, play a makeup teaching video on the display screen, respond to the user's makeup part change action, detect the user's current makeup part, and play a makeup teaching video clip corresponding to the current makeup part on the display screen, and can switch the makeup teaching video to the makeup teaching video clip corresponding to the current makeup part in real time, so that the whole process can be achieved without operating the mirror surface, and the mirror surface pollution is reduced, thereby improving the user's makeup experience.

Description

Technical Field

[0001] The present application relates to the technical field of smart home devices, and in particular to a smart mirror and a makeup teaching method. Background Art

[0002] With the development of science and technology, smart mirrors have gradually entered people's daily lives. However, when users use smart mirrors to put on makeup while watching makeup teaching videos, they need to control the video and watch the teaching of different parts. They need to use their hands to operate the mirror, and their hands may be holding cosmetics or have stains. At this time, operating the mirror will cause inconvenience, and pollute the mirror surface, affecting the imaging, giving users a bad makeup experience.

[0003] In summary, there is an urgent need for a smart mirror and a makeup teaching method to achieve the entire makeup process without operating the mirror, reduce mirror pollution, and improve the user's makeup experience. Summary of the invention

[0004] In an exemplary embodiment of the present application, a smart mirror and a makeup teaching method are provided to achieve makeup without operating the mirror, reduce mirror pollution, and improve the user's makeup experience.

[0005] In a first aspect, an exemplary embodiment of the present application provides a smart mirror, comprising:

[0006] A display screen for displaying images or videos;

[0007] A camera, used to collect a user's facial image;

[0008] The processor is configured as:

[0009] In response to the user's instruction, playing a makeup teaching video on the display screen;

[0010] In response to the user's makeup part changing action, the user's current makeup part is detected, and a makeup teaching video clip corresponding to the current makeup part is played on the display screen.

[0011] In some exemplary embodiments, the processor is configured to:

[0012] Acquire a facial image of the user captured by the camera, wherein the facial image includes a face and makeup tools;

[0013] Performing target detection on the user's facial image to determine the type and location of the user's makeup tools;

[0014] Determining whether to identify the current makeup part of the user according to the type and position of the makeup tool of the user;

[0015] If yes, in response to the current makeup part of the user, playing a makeup teaching video clip corresponding to the current makeup part on the display screen;

[0016] Otherwise, key point recognition is performed on the user's facial image to identify the key point positions of various parts of the user's face, and the attribution relationship between the key point positions of various parts of the face and the positions of the makeup tools is determined; based on the attribution relationship between the key point positions of various parts of the face and the positions of the makeup tools, the user's current makeup part is determined; in response to the user's current makeup part, a makeup teaching video clip corresponding to the current makeup part is played on the display screen.

[0017] In some exemplary embodiments, the processor is configured to:

[0018] Performing target detection on the user's facial image to detect the type and location of the user's makeup tools;

[0019] According to the position of the makeup tool, determining the center point of the position of the makeup tool;

[0020] Comparing the center point of the makeup tool with the key points of each part of the face to determine whether the center point of the makeup tool is located in an area surrounded by the key points of any part of the face;

[0021] If so, determine that the corresponding facial part among the facial parts is the current makeup part of the user; otherwise, compare the key points of the facial parts with the center point of the position of the makeup tool to determine the key point with the shortest distance from the center point of the position of the makeup tool; determine whether the distance between the key point with the shortest distance and the center point of the position of the makeup tool meets a set threshold; if the set threshold is met, determine that the facial part corresponding to the key point with the shortest distance is the current makeup part of the user; if the set threshold is not met, determine that the user's face is the current makeup part of the user.

[0022] In some exemplary embodiments, the processor is configured to:

[0023] Inputting the user's face image into a face key point detection model for detection and recognition, and identifying the key point positions of various parts of the face;

[0024] According to the key point positions of the various parts of the human face and the positions of the makeup tools, the attribution relationship between the key point positions of the various parts of the human face and the positions of the makeup tools is determined.

[0025] In some exemplary embodiments, the processor is configured to:

[0026] Acquire a first training sample, wherein the first training sample includes a face image;

[0027] Annotating the facial image to determine key point positions of various facial parts in the facial image;

[0028] Generate a labeling file corresponding to each part of the face according to the key point position of each part of the face in the face image, wherein the labeling file corresponding to each part of the face includes the key point position of each part of the face;

[0029] The convolutional neural network is trained using the facial image and the annotation files corresponding to the facial parts to obtain the facial key point detection model.

[0030] In some exemplary embodiments, the processor is configured to:

[0031] Inputting the user's face image into the target detection model for recognition, and identifying the type and location of the user's makeup tools;

[0032] The current makeup part of the user is determined according to the type and position of the makeup tools of the user.

[0033] In some exemplary embodiments, the processor is configured to:

[0034] Acquire a second training sample, wherein the second training sample includes a face image;

[0035] Annotating the facial image to determine the type and location of makeup tools in the facial image;

[0036] generating a labeling file corresponding to the makeup tool according to the type and position of the makeup tool in the face image, wherein the labeling file corresponding to the makeup tool includes the type and position of the makeup tool;

[0037] A convolutional neural network is trained using the annotation files corresponding to the facial image and the makeup tools to obtain the target detection model.

[0038] In some exemplary embodiments, the smart mirror further comprises a memory;

[0039] The processor is configured to:

[0040] Get various makeup tutorial videos;

[0041] Detect and identify each makeup teaching video, identify various makeup parts of the face in the makeup teaching video, and segment the makeup teaching video according to the various makeup parts of the face;

[0042] The segmented makeup teaching video is stored in the memory.

[0043] In a second aspect, an exemplary embodiment of the present application provides a makeup teaching method, comprising:

[0044] In response to the user's instruction, playing a makeup teaching video on the display screen;

[0045] In response to the user's makeup part changing action, the user's current makeup part is detected, and a makeup teaching video clip corresponding to the current makeup part is played on the display screen.

[0046] In the above technical solution, by responding to the user's instructions, a makeup teaching video is played on the display screen for the user to learn makeup. When the user changes his or her makeup part, the user's current makeup part can be detected in real time, and the makeup teaching video can be switched to the makeup teaching video clip corresponding to the current makeup part in real time. The user can follow the makeup teaching video continuously without any operation to complete the makeup. It is convenient and quick, and there is no need to operate the mirror throughout the process, and the mirror pollution is reduced, thereby improving the user's makeup experience.

[0047] In some exemplary embodiments, the method further comprises:

[0048] Acquire a facial image of the user captured by the camera, wherein the facial image includes a face and makeup tools;

[0049] Performing target detection on the user's facial image to determine the type and location of the user's makeup tools;

[0050] Determining whether to identify the current makeup part of the user according to the type and position of the makeup tool of the user;

[0051] If yes, in response to the current makeup part of the user, playing a makeup teaching video clip corresponding to the current makeup part on the display screen;

[0052] Otherwise, key point recognition is performed on the user's facial image to identify the key point positions of various parts of the user's face, and the attribution relationship between the key point positions of various parts of the face and the positions of the makeup tools is determined; based on the attribution relationship between the key point positions of various parts of the face and the positions of the makeup tools, the user's current makeup part is determined; in response to the user's current makeup part, a makeup teaching video clip corresponding to the current makeup part is played on the display screen.

[0053] In the above technical solution, by performing target detection on the user's face image, the type and position of the user's makeup tools are determined, and according to the type and position of the user's makeup tools, it is determined whether the user's current makeup part can be identified. If the user's current makeup part can be identified, the makeup teaching video clip corresponding to the current makeup part is played on the display screen for the user to learn makeup. If the makeup tools of different makeup parts can be similar or even the same in appearance and cannot be effectively distinguished, and the makeup part cannot be directly judged by judging the makeup tools, it is necessary to perform key point recognition on the user's face image, identify the key point positions of various parts of the user's face, and determine the attribution relationship between the key point positions of various parts of the face and the positions of the makeup tools. According to the attribution relationship between the key point positions of various parts of the face and the positions of the makeup tools, the user's current makeup part is determined, and according to the user's current makeup part, the makeup teaching video played on the display screen is switched to the makeup teaching video clip corresponding to the current makeup part in real time, so that the makeup teaching video clips are switched in real time and accurately without the user's perception, without the need for manual operation of the smart mirror, thereby improving the user's makeup experience.

[0054] In some exemplary embodiments, the method further comprises:

[0055] Performing target detection on the user's facial image to detect the type and location of the user's makeup tools;

[0056] According to the position of the makeup tool, determining the center point of the position of the makeup tool;

[0057] Comparing the center point of the makeup tool with the key points of each part of the face to determine whether the center point of the makeup tool is located in an area surrounded by the key points of any part of the face;

[0058] If so, determine that the corresponding facial part among the facial parts is the current makeup part of the user; otherwise, compare the key points of the facial parts with the center point of the position of the makeup tool to determine the key point with the shortest distance from the center point of the position of the makeup tool; determine whether the distance between the key point with the shortest distance and the center point of the position of the makeup tool meets a set threshold; if the set threshold is met, determine that the facial part corresponding to the key point with the shortest distance is the current makeup part of the user; if the set threshold is not met, determine that the user's face is the current makeup part of the user.

[0059] In the above technical solution, by comparing the center point of the makeup tool's position with the key point positions of various parts of the face, when it is determined that the center point of the makeup tool's position is located in the area surrounding the key point of any part of the face, the corresponding part of the face is determined as the user's current makeup part, otherwise the key points of various parts of the face are compared with the center point of the makeup tool's position to determine the user's current makeup part. This allows the smart mirror to switch makeup teaching video clips in real time and accurately without the user's perception, without the need for manual operation of the smart mirror, thereby improving the user's makeup experience.

[0060] In some exemplary embodiments, the method further comprises:

[0061] Inputting the user's face image into a face key point detection model for detection and recognition, and identifying the key point positions of various parts of the face;

[0062] According to the key point positions of the various parts of the human face and the positions of the makeup tools, the attribution relationship between the key point positions of the various parts of the human face and the positions of the makeup tools is determined.

[0063] In some exemplary embodiments, the method further comprises:

[0064] Acquire a first training sample, wherein the first training sample includes a face image;

[0065] Annotating the facial image to determine key point positions of various facial parts in the facial image;

[0066] Generate a labeling file corresponding to each part of the face according to the key point position of each part of the face in the face image, wherein the labeling file corresponding to each part of the face includes the key point position of each part of the face;

[0067] The convolutional neural network is trained using the facial image and the annotation files corresponding to the facial parts to obtain the facial key point detection model.

[0068] In the above technical solution, a convolutional neural network is trained with facial images and annotation files corresponding to the facial parts to obtain a facial key point detection model, so that when the user's current makeup part cannot be identified based on the type and position of makeup tools, the facial key point detection model can be used to quickly and accurately identify the key point positions of the user's facial parts, thereby identifying the user's current makeup part.

[0069] In some exemplary embodiments, the method further comprises:

[0070] Inputting the user's face image into the target detection model for recognition, and identifying the type and location of the user's makeup tools;

[0071] The current makeup part of the user is determined according to the type and position of the makeup tools of the user.

[0072] In the above technical solution, by inputting the user's facial image into the target detection model for recognition, the type and position of the user's makeup tools can be quickly and accurately identified, and the user's current makeup area can be determined based on the type and position of the user's makeup tools.

[0073] In some exemplary embodiments, the method further comprises:

[0074] Acquire a second training sample, wherein the second training sample includes a face image;

[0075] Annotating the facial image to determine the type and location of makeup tools in the facial image;

[0076] generating a labeling file corresponding to the makeup tool according to the type and position of the makeup tool in the face image, wherein the labeling file corresponding to the makeup tool includes the type and position of the makeup tool;

[0077] A convolutional neural network is trained using the annotation files corresponding to the facial image and the makeup tools to obtain the target detection model.

[0078] In the above technical solution, a convolutional neural network is trained with annotation files corresponding to facial images and makeup tools to obtain a target detection model, so that the smart mirror can detect the type and position of the user's makeup tools in real time when the user is learning makeup by referring to the makeup teaching video.

[0079] In some exemplary embodiments, the method further comprises:

[0080] Get various makeup tutorial videos;

[0081] Detect and identify each makeup teaching video, identify various makeup parts of the face in the makeup teaching video, and segment the makeup teaching video according to the various makeup parts of the face;

[0082] The segmented makeup teaching videos are stored in a memory.

[0083] In the above technical solution, the makeup teaching video is segmented according to the makeup parts of the face, and the segmented makeup teaching videos are pre-stored in the memory of the smart mirror. When the user learns to put on makeup by referring to the makeup teaching video, the makeup teaching video can be switched in real time to the makeup teaching video segment corresponding to the current makeup part in response to the user's change of makeup part. BRIEF DESCRIPTION OF THE DRAWINGS

[0084] In order to more clearly illustrate the technical solution of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0085] Figure 1 A hardware configuration block diagram of a smart mirror provided in some embodiments of the present application;

[0086] Figure 2 A software configuration block diagram of a smart mirror provided in some embodiments of the present application;

[0087] Figure 3 A schematic diagram of a user interface of a smart mirror provided in some embodiments of the present application;

[0088] Figure 4 A schematic diagram of a makeup teaching method provided in some embodiments of the present application;

[0089] Figure 5 A schematic diagram of a process for identifying various makeup parts of a face in a makeup teaching video provided in some embodiments of the present application;

[0090] Figure 6 A schematic diagram of a process for identifying makeup areas on a user's face provided in some embodiments of the present application;

[0091] Figure 7 A schematic diagram of a process for determining the relationship between the key point positions of various parts of a face and the positions of makeup tools provided in some embodiments of the present application;

[0092] Figure 8 A schematic diagram of a page for switching makeup instruction videos in a smart mirror provided in some embodiments of the present application. DETAILED DESCRIPTION

[0093] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present application.

[0094] Figure 1 The hardware configuration block diagram of a smart mirror provided by an embodiment of the present application is shown as an example. Figure 1 As shown, the smart mirror 100 includes a processor 110 , a detector 120 , a communication interface 130 , a display screen 140 , an illuminating lamp 150 , an audio output interface 160 , a memory 170 , and a power supply 180 .

[0095] The processor 110 includes a CPU processor 111, a RAM 112, a ROM 113, a graphics processor 114, a communication interface 115, a video processor 116, an audio processor 117, and a communication bus. Among them, the RAM 112 and the ROM 113 and the CPU processor 111, the graphics processor 114, the communication interface 115, the video processor 116, and the audio processor 117 are connected through the communication bus; the communication interface 115 may include a first interface 115-1 to an nth interface 115-n. These interfaces may also be network interfaces connected to external devices via a network.

[0096] ROM 113 is used to store various system startup instructions. For example, when a power-on signal is received, the power of the smart mirror 100 starts to start, and the CPU processor 111 runs the system startup instructions in the ROM, and copies the operating system stored in the memory 170 to the RAM 112 to start running the operating system. After the operating system is started, the CPU processor 111 copies various application programs in the memory 170 to the RAM 112, and then starts running various application programs.

[0097] The graphics processor 114 is used to generate various graphic objects, such as icons, operation menus, and user input command display graphics. It includes an operator, which performs operations by receiving various interactive instructions input by the user and displays various objects according to display attributes. It also includes a renderer, which generates various objects based on the operator, and displays the rendered results on the display screen 140.

[0098] The CPU processor 111 is used to execute the operating system and application instructions stored in the memory 170, and execute various applications, data and content according to various interactive instructions received from the outside, so as to finally display and play various audio and video contents.

[0099] In some exemplary embodiments, the CPU processor 111 may include multiple processors. The multiple processors may include a main processor and multiple or one sub-processors. The main processor is used to perform some operations of the smart mirror 100 in the pre-power-on mode and / or the operation of displaying an image in the normal mode. Multiple or one sub-processors are used for an operation in the standby mode and the like.

[0100] The video processor 116 is used to receive an external video signal and perform video processing such as decompression, decoding, scaling, noise reduction, frame rate conversion, resolution conversion, and image synthesis according to the standard codec protocol of the input signal to obtain a signal that can be directly displayed or played on the display screen 140.

[0101] For example, the video processor 116 includes a demultiplexing module, a video decoding module, an image synthesis module, a frame rate conversion module, a display formatting module, and the like.

[0102] The demultiplexing module is used to demultiplex the input audio and video data streams. For example, if MPEG-2 is input, the demultiplexing module demultiplexes the data streams into video signals and audio signals.

[0103] The video decoding module is used to process the demultiplexed video signal, including decoding and scaling.

[0104] The image synthesis module, such as an image synthesizer, is used to superimpose and mix the GUI signal generated by the graphics generator according to the user input or itself with the scaled video image to generate an image signal for display.

[0105] The frame rate conversion module is used to convert the input video frame rate, such as converting the 60Hz frame rate to 120Hz frame rate or 240Hz frame rate. The usual format is implemented by interpolation.

[0106] The display formatting module is used to receive the frame rate converted video output signal and change the signal to conform to the display format signal, such as outputting RGB data signal.

[0107] The audio processor 117 is used to receive external audio signals, and perform decompression and decoding, as well as noise reduction, digital-to-analog conversion and amplification processing according to the standard codec protocol of the input signal to obtain a sound signal that can be played in the speaker.

[0108] In some other exemplary embodiments, the video processor 116 may include one or more chips. The audio processor 117 may also include one or more chips. And in some other exemplary embodiments, the video processor 116 and the audio processor 117 may be separate chips or integrated with the processor 110 in one or more chips.

[0109] The detector 120 is used by the smart mirror 100 to collect signals of the external environment or interact with the outside. The detector 120 includes an image collector 121, such as a camera, a webcam, etc., which can be used to collect external environment scenes, as well as user attributes or user face images.

[0110] In some other exemplary embodiments, the detector 120 may also include a sound collector 122, such as a microphone, which can be used to receive the user's voice, including the voice signal of the user's control instructions for controlling the smart mirror 100, or collect environmental sounds for identifying the type of environmental scene.

[0111] In some other exemplary embodiments, the detector 120 may also include a weather collector 123, such as a temperature detector, for collecting current weather temperature, or collecting weather and climate attribute data such as the current season.

[0112] The communication interface 130 is a component for communicating with external devices or external servers according to various communication protocol types. For example, the communication interface 130 can be a Wifi module 131, a Bluetooth module 132, a wired Ethernet module 133, a USB 134, or other network communication protocol modules or near field communication protocol modules.

[0113] The smart mirror 100 can establish control signal and data signal transmission and reception with an external control device or a content providing device through the communication interface 130 .

[0114] The display screen 140 is used to receive the image signal input from the video processor 116 to display the video content, image and menu control interface. The display screen 140 includes a display screen component for presenting images and a driving component for driving the image display. The displayed video content can come from the video content stored in the memory 170, or from the video content input from the communication interface 130. In addition, the display screen 160 displays the user control UI interface generated in the smart mirror 100 and used to control the smart mirror 100; the user can also input the user's operation command in the graphical user interface (GUI) displayed on the display screen 140.

[0115] The lighting lamp 150 is used to provide supplementary light for the user when learning to put on makeup using the smart mirror 100.

[0116] The audio output interface 160 is used to receive the audio signal output by the audio processor 117. The audio output interface 160 may include a built-in speaker 161 (such as a horn), or an external audio output terminal 162 of a generating device that outputs to an external device, such as an external audio interface, a headphone interface, etc.

[0117] The memory 170 is used to store various operating programs, data and applications that drive and control the smart mirror 100. The memory 170 can store various control signal instructions input by the user. It includes storing various software modules for driving the smart mirror 100. For example, the various software modules stored in the memory 170 include: a basic module, a detection module, a display control module, a communication module, etc.

[0118] Among them, the basic module is a bottom-level software module used for signal communication between various hardware in the smart mirror 100 and sending processing and control signals to the upper-level module; the detection module is a management module used to collect various information from various detectors or user input interfaces, and perform digital-to-analog conversion and analysis management; the display control module is a module used to control the display screen 140 to display image content, and can be used to play multimedia image content and UI interface information. The communication module is a module used for control and data communication with external devices.

[0119] At the same time, the memory 170 can also be used to store received external data and user data, images in various user interfaces, visual effect diagrams, and the like.

[0120] In addition, the memory 170 is specifically used to store the operating program that drives the processor 110 in the smart mirror 100, and to store various applications built into the smart mirror 100, various applications downloaded by users from external devices, various graphical user interfaces related to the applications, various objects related to the graphical user interfaces, user data information, and various internal data supporting the applications. The memory 170 is also used to store system software such as the OS kernel, middleware, and applications, as well as to store drivers and related data such as the display screen 140, the communication interface 130, and the input / output interface of the detector 120, or to store other user data.

[0121] The power supply 180 is used to provide power support for the startup and operation of each component in the smart mirror 100. It can be in the form of a battery and related control circuits. Under the operation of the user, the power input from the external power supply provides power supply support for the smart mirror 100. The power supply 180 may include a built-in power supply circuit installed inside the smart mirror 100, or it may be a power supply installed outside the smart mirror 100, providing a power interface for an external power supply in the smart mirror 100.

[0122] Figure 2 The following is an exemplary diagram of a software configuration block diagram of a smart mirror provided in an embodiment of the present application. Figure 2 As shown, it may include an operating system 171 , an interface layout manager 172 , an event transmission system 173 , and an application 174 .

[0123] The operating system 171 includes execution software for processing various basic system services and for implementing hardware-related tasks, and acts as a medium for data processing between applications and hardware components, such as the Android operating system. In some embodiments, part of the operating system kernel may include a series of software to manage the hardware resources of the smart mirror 100 and provide services for other programs or software codes.

[0124] In some other embodiments, part of the operating system kernel may include one or more device drivers, which may be a set of software codes in the operating system that help operate or control the devices or hardware associated with the smart mirror. The driver may include code to operate video, audio and / or other multimedia components. Examples include display screens, cameras, Flash, and WiFi.

[0125] The accessibility module 1711 is used to access or modify the application program to achieve the accessibility of the application program and the operability of its displayed content.

[0126] The communication module 1712 is used to connect to other peripheral devices via relevant communication interfaces and communication networks.

[0127] The user interface module 1713 is used to provide objects for displaying user interfaces so that each application can access them and realize user operability, such as the front-end interactive interface of a smart mirror.

[0128] Control application 1714 is used for controllable process management, including runtime applications, etc.

[0129] The event transmission system 173 can be implemented in the operating system 171 or in the application 174. In some embodiments, it is implemented in the operating system 171 on the one hand, and in the application 174 on the other hand, and is used to monitor various user input events and implement one or more groups of predefined operation handlers based on the recognition results of various events or sub-events in response to various event references.

[0130] The event monitoring module 1731 is used to monitor the input events or sub-events of the user input interface; the event identification module 1732 is used to define various events input by various user input interfaces, identify various events or sub-events, and transmit them to the processor 110 for executing one or more corresponding processing programs. For example, the processor 110 processes the corresponding events or sub-events according to the logic program and core algorithm stored in the smart mirror 100, and presents the processed results on the display screen 140.

[0131] Among them, an event or sub-event refers to an input detected by one or more detectors in the smart mirror 100, such as various sub-events of user voice input or various sub-events of display screen operation input.

[0132] The interface layout manager 172 directly or indirectly receives events or sub-events from the event transmission system 173 that monitor each user input, and is used to update the layout of the user interface, including but not limited to the position of each control or sub-control in the interface, as well as the size or position of the container, the hierarchy, and other execution operations related to the interface layout.

[0133] Figure 3 The following is a schematic diagram of a user interface of a smart mirror provided by an embodiment of the present application. Figure 3 As shown, the user interface includes multiple view display areas. For example, when the user clicks on the makeup teaching video that he wants to learn, the smart mirror automatically splits the screen into two upper and lower areas, namely the first view display area 301 and the second view display area 302. The first view display area 301 is used for the user to look in the mirror and put on makeup, and the second view display area 302 is used to play the makeup teaching video that the user is learning. Among them, the display screen (such as the mirror) of the smart mirror can be in full-screen mode, so that the user can browse the corresponding beauty mall, or it can also be in full-mirror mode, so that the user can look in the mirror or put on makeup for other purposes, or it can be in half-mirror mode, so that the user can learn makeup while watching the makeup teaching video or for other purposes; each view display area includes one or more different items, and the user interface also includes a selector indicating that any item is selected.

[0134] It should be noted that multiple view display areas can be visible boundaries or invisible boundaries. For example, different view display areas can be identified by different background colors of each view display area, or by visible identification such as boundary lines, or by invisible invisible boundaries. There may also be no visible or non-visible boundaries, and only related views in a certain range area are displayed on the screen. When the size and / or arrangement of the same change attributes are changed, the certain range area is regarded as the existence of a boundary of a view partition.

[0135] In the smart mirror provided by some embodiments of the present application, the processor may be configured to: respond to the user's instructions, play the makeup teaching video on the display screen, respond to the user's makeup part change action, detect the user's current makeup part, and play the makeup teaching video clip corresponding to the current makeup part on the display screen. For example, if the user wants to learn a makeup video for a date, the corresponding date makeup teaching video can be determined according to the date's makeup through the operation instruction triggered by the display screen or the voice instruction issued by the voice device, and the makeup teaching video is played on the display screen 140 on the smart mirror. When the smart mirror recognizes that the user changes the makeup part, the makeup teaching video is immediately switched to the makeup teaching video clip corresponding to the current makeup part for the user to watch and learn, and compare the makeup.

[0136] Optionally, the processor is further configured to: obtain a facial image of the user captured by a camera, the facial image including a face and makeup tools, perform target detection on the facial image of the user, determine the type and position of the user's makeup tools, and determine whether the user's current makeup part is recognized based on the type and position of the user's makeup tools; if so, in response to the user's current makeup part, play a makeup teaching video clip corresponding to the current makeup part on the display screen; otherwise, perform key point recognition on the facial image of the user, identify the key point positions of various parts of the user's face, and determine the attribution relationship between the key point positions of various parts of the face and the positions of makeup tools; determine the user's current makeup part based on the attribution relationship between the key point positions of various parts of the face and the positions of makeup tools; and in response to the user's current makeup part, play a makeup teaching video clip corresponding to the current makeup part on the display screen. For example, when a user is learning a makeup tutorial video for a date, the user's current makeup part is identified based on the type and position of the user's makeup tools, and a makeup tutorial video clip corresponding to the current makeup part is played on the display screen. If the appearance of makeup tools for different makeup parts can be similar or even the same, for example, an eyebrow pencil is used when drawing eyebrows, and an eyeliner is used when drawing eyeliner, and the eyebrow pencil and eyeliner are similar and cannot be effectively distinguished, and the makeup part cannot be directly judged by judging the makeup tools, it is necessary to perform key point recognition on the user's facial image, identify the key point positions of various parts of the user's face, and determine the attribution relationship between the key point positions of various parts of the face and the positions of makeup tools. According to the attribution relationship between the key point positions of various parts of the face and the positions of makeup tools, the user's current makeup part is determined, and in response to the user's current makeup part, a makeup tutorial video clip corresponding to the current makeup part is played on the display screen.

[0137] Optionally, the processor is further configured to: obtain a facial image of the user captured by a camera, the facial image including a face and makeup tools, perform target detection on the facial image of the user, determine the type and position of the user's makeup tools, and determine whether the user's current makeup part is recognized based on the type and position of the user's makeup tools; if so, in response to the user's current makeup part, play a makeup teaching video clip corresponding to the current makeup part on the display screen; otherwise, perform key point recognition on the facial image of the user, identify the key point positions of various parts of the user's face, and determine the attribution relationship between the key point positions of various parts of the face and the positions of makeup tools; determine the user's current makeup part based on the attribution relationship between the key point positions of various parts of the face and the positions of makeup tools, and play a makeup teaching video clip corresponding to the current makeup part on the display screen in response to the user's current makeup part.

[0138] Optionally, the processor is further configured to: perform target detection on the user's facial image, detect the type and position of the user's makeup tools, determine the center point of the makeup tool position based on the position of the makeup tool, compare the center point of the makeup tool position with the key point positions of various facial parts, and determine whether the center point of the makeup tool position is located in the area surrounded by the key points of any facial part; if so, determine that the corresponding facial part among the facial parts is the user's current makeup part; otherwise, compare the key points of various facial parts with the center point of the makeup tool position, determine the key point with the shortest distance from the center point of the makeup tool position, and determine whether the distance between the key point with the shortest distance and the center point of the makeup tool position meets a set threshold; if so, determine that the facial part corresponding to the key point with the shortest distance is the user's current makeup part; if not, determine that the user's face is the user's current makeup part.

[0139] Optionally, the processor is also configured to: input the user's facial image into a facial key point detection model for detection and recognition, identify the key point positions of various parts of the face, and determine the attribution relationship between the key point positions of various parts of the face and the positions of makeup tools based on the key point positions of various parts of the face and the positions of makeup tools.

[0140] Optionally, the processor is also configured to: obtain a first training sample, the first training sample includes a face image, annotate the face image, determine the key point positions of each face part in the face image, generate an annotation file corresponding to each face part according to the key point positions of each face part in the face image, the annotation file corresponding to each face part includes the key point positions of each face part, train a convolutional neural network with the face image and the annotation files corresponding to each face part, and obtain a face key point detection model.

[0141] Optionally, the processor is further configured to: input the user's facial image into the target detection model for recognition, identify the type and position of the user's makeup tools, and determine the user's current makeup area based on the type and position of the user's makeup tools.

[0142] Optionally, the processor is further configured to: obtain a second training sample, the second training sample includes a face image; annotate the face image, determine the type and position of makeup tools in the face image, generate an annotation file corresponding to the makeup tools according to the type and position of the makeup tools in the face image, the annotation file corresponding to the makeup tools includes the type and position of the makeup tools, train the convolutional neural network with the face image and the annotation file corresponding to the makeup tools, and obtain a target detection model.

[0143] Optionally, the processor is also configured to: obtain various makeup teaching videos, detect and identify each makeup teaching video, identify the various makeup parts of the face in the makeup teaching video, and segment the makeup teaching video according to the various makeup parts of the face, and store the segmented makeup teaching videos in a memory.

[0144] Figure 4 The flowchart of a makeup teaching method is shown as an example. The flowchart can be executed by the smart mirror 100.

[0145] like Figure 4 As shown, the process includes:

[0146] Step 401, in response to a user's instruction, playing a makeup teaching video on a display screen.

[0147] In the embodiment of the present application, a makeup teaching video is played on the display screen in response to the user's instruction. According to the makeup teaching video that the user wants to learn, the makeup teaching video is played in the corresponding display screen area on the smart mirror for the user to learn makeup. The types of makeup teaching videos may include dating, shopping, attending wedding banquets or interviews, etc.

[0148] Since both the real-time detected user face and the makeup teaching video provided by the beauty content provider or available from the Internet need to identify the makeup parts of the face, in order to ensure that only the makeup user who is detected in real time can recognize the makeup parts of the face in real time, before playing the makeup teaching video on the display screen, it is necessary to first identify the makeup parts of the face in the makeup teaching video, and divide the makeup teaching video according to the makeup parts of the face and store it in the memory of the smart mirror in advance. Figure 5 The following is a schematic diagram of a process for identifying various makeup parts of a face in a makeup teaching video. The process can be executed by the smart mirror 100.

[0149] like Figure 5 As shown, the process includes:

[0150] Step 501, obtaining a face image in a makeup teaching video.

[0151] In the embodiment of the present application, the face image in the makeup teaching video may refer to a face image in any makeup teaching video, and the face image may include a face and makeup tools. The makeup teaching video may be any makeup teaching video, and the makeup teaching video is provided by a beauty content provider or can be obtained from the Internet.

[0152] Step 502: input the face image into the target detection model for processing.

[0153] In an embodiment of the present application, a face image is input into a target detection model for processing to obtain the type and position of makeup tools in the face image. Specifically, the target detection model is constructed based on a convolutional neural network. A large number of face images containing makeup tools are first obtained from the Internet and reality. The makeup tools in the face images are annotated by a labeling method, and a corresponding annotation file containing the annotation coordinates and the type of makeup tools is generated. The face image and the corresponding annotation file are used as training data for the convolutional neural network. The convolutional neural network learns the position and features of makeup tools in the face image by performing a series of calculation operations on the input face image and the annotation information. After the training is completed, a target detection model is generated. A new face image containing makeup tools is input into the target detection model, and the target detection model automatically outputs the position and type of makeup tools in the face image.

[0154] Step 503, obtaining the type and position of the makeup tools in the face image.

[0155] Step 504, determining whether the current makeup part in the makeup teaching video is recognized.

[0156] In the embodiment of the present application, it is determined whether the current makeup part in the makeup teaching video is recognized. If so, step 508 is executed, and if not, step 505 is executed.

[0157] Step 505: input the face image into the face key point detection model for processing.

[0158] In an embodiment of the present application, after determining that the current makeup part in the makeup teaching video cannot be identified, the face image is input into the face key point detection model for processing to obtain the key point positions of each part of the face. Specifically, since the appearance of makeup tools for different makeup parts can be similar or even the same, such as using an eyebrow pencil when drawing eyebrows and using an eyeliner when drawing eyeliner, eyebrow pencils and eyeliners are similar and cannot be effectively distinguished, the makeup part cannot be directly judged by judging the makeup tools, so the face key point detection model is used to detect the face image during the makeup process. Each key point has its own fixed type, such as eyes, mouth, eyebrows, etc., and contains position coordinate information. The face key point detection model is constructed based on a convolutional neural network. First, a large number of face images are obtained from the network and reality, and each part of the face on the face image is punctuated by a labeling method, such as eyes, nose, mouth, eyebrows, etc. The number of points for each part is fixed, and a corresponding labeling file containing key point position coordinate information is generated during labeling. The labeled face images and labeled files are input into the convolutional neural network as training data. The convolutional neural network learns the position characteristics of the key points in the face images through a series of calculation operations. After the training is completed, a face key point detection model is generated. The new face image is input into the face key point detection model, and the face key point detection model automatically outputs the key point positions of each part of the face.

[0159] Step 506, obtaining the key point positions of each part of the face.

[0160] Step 507, determining the relationship between the key point positions of various parts of the face and the positions of the makeup tools.

[0161] In an embodiment of the present application, based on the key point positions of various parts of the face and the positions of makeup tools, the attribution relationship between the key point positions of various parts of the face and the positions of makeup tools is determined, and based on the attribution relationship between the key point positions of various parts of the face and the positions of the makeup tools, the current makeup part in the makeup teaching video is determined.

[0162] Step 508, determining the current makeup part in the makeup teaching video.

[0163] In an embodiment of the present application, by performing target detection on a face image or performing facial key point detection, the current makeup part in the makeup teaching video can be determined.

[0164] It should be noted that, according to this process, the various makeup parts of the face can be identified for each makeup teaching video, and the makeup teaching video can be segmented according to the various makeup parts of the face, and then the segmented makeup teaching videos can be pre-stored in the memory of the smart mirror.

[0165] Step 402, in response to the user's makeup part changing action, detecting the user's current makeup part, and playing a makeup teaching video clip corresponding to the current makeup part on the display screen.

[0166] In an embodiment of the present application, when the smart mirror recognizes that the user changes the makeup part, the makeup teaching video is immediately switched to the makeup teaching video clip corresponding to the current makeup part in real time.

[0167] After the makeup teaching video is segmented according to the makeup parts of the face and the segmented makeup teaching videos are pre-stored in the memory of the smart mirror, when the user learns to put on makeup by referring to the makeup teaching video, the user's current makeup part is detected in response to the user's change of makeup part, and the makeup teaching video clip corresponding to the current makeup part is played on the display screen. Figure 6 The following is a schematic diagram of a process for identifying makeup areas on a user's face. The process can be executed by the smart mirror 100.

[0168] like Figure 6 As shown, the process includes:

[0169] Step 601, obtaining a user's face image.

[0170] In the embodiment of the present application, the face image of the user may include the face and makeup tools. The face image is captured by the camera of the smart mirror when the user learns to put on makeup by referring to the makeup teaching video. When the smart mirror detects the face of the user, it obtains the video stream resource, converts the video stream resource into image resource frame by frame, and then inputs it into the target detection model for detection and recognition.

[0171] Step 602: Input the user's face image into the target detection model for detection.

[0172] In an embodiment of the present application, the user's facial image is input into the target detection model for detection, and the type and position of the user's makeup tools can be identified. For example, for the identification of certain makeup parts, based on the specificity of the makeup tools, the makeup parts can be identified directly by identifying the makeup tools. For example, when using a makeup egg or a powder puff for makeup, it is generally a large-scale concealer and foundation application on the face; when using lipstick, it is generally a coloring operation on the lips. Therefore, the target detection model can be used to detect and identify makeup tools such as makeup eggs, powder puffs, and lipsticks. When the relevant makeup tools are detected in the video screen for 1s, the user's current makeup part can be determined, and the makeup teaching video will immediately jump to the video clip corresponding to the makeup part where the user is making up, so that the user can watch the makeup teaching video of the makeup part and learn makeup by following the makeup teaching video.

[0173] Step 603, obtaining the type and position of the makeup tools in the face image.

[0174] Step 604, determining whether the user's current makeup area is recognized.

[0175] In the embodiment of the present application, it is determined whether the current makeup part of the user is recognized according to the type and position of the makeup tool of the user. If so, step 608 is executed, and if not, step 605 is executed.

[0176] Step 605: input the user's face image into the face key point detection model for detection.

[0177] In an embodiment of the present application, when it is determined that the user's current makeup area cannot be recognized, the user's facial image is input into a facial key point detection model for detection to identify the key point positions of various facial parts.

[0178] Step 606, obtaining the key point positions of each part of the face.

[0179] Step 607, determining the relationship between the key point positions of each part of the face and the positions of the makeup tools.

[0180] In the embodiments of the present application, since the appearance of makeup tools for different makeup parts can be similar or even identical, the makeup parts cannot be directly judged by judging the makeup tools. It is necessary to perform key point recognition on the user's facial image, identify the key point positions of various parts of the user's face, and determine the relationship between the key point positions of various parts of the face and the positions of makeup tools. Figure 7 The flowchart of judging the relationship between the key points of various parts of the face and the positions of makeup tools is shown as an example. The flowchart can be executed by the smart mirror 100.

[0181] like Figure 7 As shown, the process includes:

[0182] Step 701, obtaining the center point of the makeup tool.

[0183] In the embodiment of the present application, the position of the makeup tool is analyzed and the center point of the position of the makeup tool is calculated.

[0184] Step 702, obtaining the key point surrounding area of ​​each part of the face.

[0185] In the embodiment of the present application, the positions of key points of various parts of the face are analyzed to determine the areas surrounding the key points of various parts of the face.

[0186] Step 703: determine whether the center point of the makeup tool is located in an area surrounding a key point of any facial part.

[0187] In the embodiment of the present application, the center point of the makeup tool is compared with the key points of each part of the face to determine whether the center point of the makeup tool is located in the area surrounding the key points of any part of the face. If so, step 704 is executed, and if not, step 705 is executed.

[0188] Step 704: determine the corresponding facial part among the facial parts as the current makeup part of the user.

[0189] In the embodiment of the present application, if the center point of the position of the makeup tool is determined to be located in the area surrounding the key points of any facial part among the facial parts, then the facial part is determined to be the user's current makeup part.

[0190] Step 705, calculate the key point with the shortest distance from the center point of the makeup tool.

[0191] In the embodiment of the present application, if it is determined that the center point of the makeup tool is not located in the area surrounded by the key points of any facial part, the key point with the shortest distance from the center point of the makeup tool is calculated.

[0192] Step 706 , determining whether the distance between the key point with the smallest distance and the center point of the makeup tool position meets a set threshold.

[0193] In the embodiment of the present application, it is determined whether the distance between the key point with the smallest distance and the center point of the makeup tool meets the set threshold. If so, step 707 is executed, and if not, step 708 is executed. The set threshold can be set based on experience.

[0194] Step 707, determining the facial part corresponding to the key point with the smallest distance as the user's current makeup part.

[0195] In an embodiment of the present application, when the distance between the key point with the smallest distance and the center point of the position of the makeup tool meets the set threshold, the facial part corresponding to the key point with the smallest distance is determined to be the user's current makeup part.

[0196] Step 708: determine the user's face as the user's current makeup area.

[0197] In the embodiment of the present application, when the distance between the key point with the smallest distance and the center point of the position of the makeup tool does not meet the set threshold, the user's face is determined to be the user's current makeup part.

[0198] It should be noted that while detecting the key points of the user's face, the target detection model is used to detect and identify the makeup tools in the face image, including the makeup pen head, the makeup brush head, etc., and the makeup tools are detected to obtain the location information and category information of the makeup tools. After obtaining the location information of the key points of the face and the location information of the makeup tools, the two types of location information are analyzed in the same face image. First, the center point of the makeup tool is calculated, and then compared with the location information of the key points of the face. If the center point of the makeup tool falls within the area surrounded by the key points of a certain part of the face, it is determined that the user's current makeup part is this part; if the center point of the makeup tool does not fall within the area surrounded by the key points of a certain part of the face, the face key point closest to the center point of the makeup tool is calculated. If the distance between the two points is within a set threshold, the part information of the face key point is obtained, and the user's current makeup part is determined to be this part. If the distance between the two points is beyond a set threshold, the user's current makeup part is determined to be the face.

[0199] Step 608, determining the user's current makeup area.

[0200] In an embodiment of the present application, the user's current makeup area can be determined by performing target detection or facial key point detection on the user's facial image.

[0201] Step 609, play the makeup teaching video clip corresponding to the current makeup part.

[0202] In the embodiment of the present application, in response to the user's makeup part change action, the user's current makeup part is detected, and the makeup teaching video clip corresponding to the current makeup part is played on the display screen. For example, when the smart mirror recognizes that the user has changed the makeup part, the makeup teaching video is immediately switched to the makeup teaching video clip corresponding to the current makeup part in real time. The page where the smart mirror switches the makeup teaching video can be as follows: Figure 8 shown.

[0203] It can be seen from the above technical scheme that the present application provides a smart mirror and a makeup teaching method, which can play a makeup teaching video on a display screen for users to learn makeup by responding to user instructions. When the user changes his or her makeup part, the user's current makeup part can be detected in real time, and the makeup teaching video can be switched to the makeup teaching video clip corresponding to the current makeup part in real time. The user can follow the makeup teaching video continuously without any operation to complete the makeup, which is convenient and quick, and can achieve the whole process without operating the mirror, reduce mirror pollution, and improve the user's makeup experience.

[0204] Since the communication terminal and computer storage medium in the embodiments of the present application can be applied to the above-mentioned processing method, the technical effects that can be obtained can also refer to the above-mentioned method embodiments, and the embodiments of the present application will not be repeated here.

[0205] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0206] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0207] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0208] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0209] Although the preferred embodiments of the present application have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0210] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A smart mirror, characterized in that: include: A display screen for displaying images or videos; A camera, used to collect a user's facial image; The processor is configured as: In response to the user's instruction, playing a makeup teaching video on the display screen; In response to the transformation action of the makeup part of the user, the current makeup part of the user is detected, and a makeup teaching video clip corresponding to the current makeup part is played on the display screen; wherein the processor is configured to: obtain the face image of the user captured by the camera, the face image including the face and makeup tools; perform target detection on the face image of the user to determine the position of the makeup tools of the user; perform key point recognition on the face image of the user to identify the key point positions of each part of the face of the user; determine the center point of the position of the makeup tool according to the position of the makeup tool; compare the center point of the position of the makeup tool with the key point positions of each part of the face to determine whether the center point of the position of the makeup tool is located in the area surrounded by the key points of any part of the face; if so, determine that the corresponding part of the face among the parts of the face is the current makeup part of the user; otherwise, compare the key points of each part of the face with the center point of the position of the makeup tool to determine the key point with the smallest distance from the center point of the position of the makeup tool; Determine whether the distance between the key point with the smallest distance and the center point of the position of the makeup tool meets a set threshold; if the set threshold is met, determine that the facial part corresponding to the key point with the smallest distance is the current makeup part of the user; if the set threshold is not met, determine that the user's face is the current makeup part of the user.

2. The smart mirror according to claim 1, characterized in that: The processor is configured to: Performing target detection on the user's facial image to determine the type and location of the user's makeup tools; If the current makeup part of the user is determined to be identified based on the type and position of the user's makeup tools, a makeup teaching video clip corresponding to the current makeup part is played on the display screen in response to the current makeup part of the user.

3. The smart mirror according to claim 1, characterized in that: The processor is configured to: Inputting the user's face image into a face key point detection model for detection and recognition, and identifying the key point positions of various parts of the face; According to the key point positions of the various parts of the human face and the positions of the makeup tools, the attribution relationship between the key point positions of the various parts of the human face and the positions of the makeup tools is determined.

4. The smart mirror according to claim 3, characterized in that: The processor is configured to: Acquire a first training sample, wherein the first training sample includes a face image; Annotating the facial image to determine key point positions of various facial parts in the facial image; Generate a labeling file corresponding to each part of the face according to the key point position of each part of the face in the face image, wherein the labeling file corresponding to each part of the face includes the key point position of each part of the face; The convolutional neural network is trained using the facial image and the annotation files corresponding to the facial parts to obtain the facial key point detection model.

5. The smart mirror according to claim 2, characterized in that: The processor is configured to: Inputting the user's face image into the target detection model for recognition, and identifying the type and location of the user's makeup tools; The current makeup part of the user is determined according to the type and position of the makeup tools of the user.

6. The smart mirror according to claim 5, characterized in that: The processor is configured to: Acquire a second training sample, wherein the second training sample includes a face image; Annotating the facial image to determine the type and location of makeup tools in the facial image; generating a labeling file corresponding to the makeup tool according to the type and position of the makeup tool in the face image, wherein the labeling file corresponding to the makeup tool includes the type and position of the makeup tool; A convolutional neural network is trained using the annotation files corresponding to the facial image and the makeup tools to obtain the target detection model.

7. The smart mirror according to any one of claims 1 to 6, characterized in that: The smart mirror also includes a memory; The processor is configured to: Get various makeup tutorial videos; Detect and identify each makeup teaching video, identify various makeup parts of the face in the makeup teaching video, and segment the makeup teaching video according to the various makeup parts of the face; The segmented makeup teaching video is stored in the memory.

8. A makeup teaching method, characterized in that: include: In response to the user's instruction, playing a makeup teaching video on the display screen; In response to the user's makeup part transformation action, the user's current makeup part is detected, and a makeup teaching video clip corresponding to the current makeup part is played on the display screen; wherein, in response to the user's makeup part transformation action, the user's current makeup part is detected, including: acquiring a face image of the user, the face image including a face and makeup tools; performing target detection on the user's face image to determine the position of the user's makeup tools; performing key point recognition on the user's face image to identify the key point positions of various parts of the user's face; determining the center point of the makeup tool position according to the position of the makeup tool; comparing the center point of the makeup tool position with the key point of various parts of the face The key point positions are compared to determine whether the center point of the position of the makeup tool is located in the area surrounded by the key points of any of the face parts; if so, the corresponding face part of the face parts is determined to be the current makeup part of the user; otherwise, the key points of the face parts are compared with the center point of the position of the makeup tool to determine the key point with the smallest distance from the center point of the position of the makeup tool; determine whether the distance between the key point with the smallest distance and the center point of the position of the makeup tool meets a set threshold; if the set threshold is met, the face part corresponding to the key point with the smallest distance is determined to be the current makeup part of the user, and if the set threshold is not met, the user's face is determined to be the current makeup part of the user.

9. The method according to claim 8, characterized in that The method further comprises: Performing target detection on the user's facial image to determine the type and location of the user's makeup tools; If the current makeup part of the user is determined to be identified based on the type and position of the user's makeup tools, a makeup teaching video clip corresponding to the current makeup part is played on the display screen in response to the current makeup part of the user.

Images