Method and electronic device for displaying skin details in augmented reality

Abstract

The present application provides a method and electronic device for displaying skin details in an augmented reality manner, wherein the method for displaying skin details in an augmented reality manner is applied to an electronic device, and the method includes: acquiring a facial image; determining image features at a specific position in the facial image; acquiring a skin state corresponding to the image features at the specific position; activating enhanced content corresponding to the skin state according to a preset operation for the specific position, and displaying the enhanced content superimposed on the facial image. According to the method of the embodiment of the present application, the enhanced content can be combined with the facial image, the skin details can be magnified, and the internal structure of various skin problems can be displayed, so that the user can understand the root cause of the skin problem, learn skin-related knowledge more deeply, and increase the fun of user use.

Description

Technical Field

[0001] The present application relates to the field of augmented reality display technology, and in particular to a method, device, electronic device and computer-readable storage medium for skin detail magnification augmented reality display. Background Art

[0002] Augmented Reality (AR) is a technology that calculates the position and angle of camera images in real time and adds corresponding images. The goal of this technology is to overlay the virtual world on the real world on the screen and interact with it. With the improvement of the computing power of portable electronic products, the use of augmented reality is becoming more and more extensive.

[0003] Currently, there are many skin analysis applications, and even dedicated skin analysis instruments. Such skin analysis research also uses augmented reality technology to show users the details of the skin surface. In addition to photos of skin surface details, users have more needs for some additional information, including the causes and solutions of skin problems. Summary of the invention

[0004] In view of this, the present application provides a method, device, electronic device and computer-readable storage medium for skin detail magnification and augmented reality display, which can enhance the reality display of the subcutaneous structure model corresponding to skin problems, so that the user can intuitively see the subcutaneous tissue structure corresponding to the skin problem, so as to increase the fun and meet the user's exploration needs for the subcutaneous tissue structure corresponding to the skin problem.

[0005] Some embodiments of the present application provide a method for skin detail magnification augmented reality display. The present application is introduced from multiple aspects below, and the embodiments and beneficial effects of the following multiple aspects can be referenced to each other.

[0006] In the first aspect, the present application provides a method for skin detail magnification augmented reality display, which is applied to an electronic device, the electronic device includes a display screen, and the method includes: the electronic device obtains a face image to be analyzed, and displays the face image on the display screen of the electronic device, for example, the face image can be obtained through an image acquisition function such as a mobile terminal, and a shooting function, and the obtained face image is displayed on the display screen of a smart phone; the electronic device determines the image features of a specific position in the face image, wherein the image features can be color features, pattern outlines, color density, or a combination of the two or more features at the specific position; the electronic device obtains the skin state corresponding to the image features at the specific position, wherein the skin state can be blackheads, acne, and moles; the electronic device activates the enhanced content corresponding to the skin state according to the preset operation for the specific position, so as to display the enhanced content on the display screen. Wherein, the enhanced content is pre-stored, including virtual content, or a combination of virtual content and real content. By superimposing the skin state of the preset position and the enhanced content on the display screen, the user's interest in use is improved and the user experience is improved.

[0007] In a possible implementation of the first aspect above, an electronic device determines image features at a specific position in a facial image, including: determining a region of interest in the facial image, determining image features at the specific position in the region of interest, and by further determining the image features at the specific position within the region of interest, the accuracy of image feature analysis can be improved.

[0008] In a possible implementation of the first aspect above, determining the region of interest on a facial image includes: detecting key points of a face in the facial image, wherein the key points include: the positions of the contour of the face, eyebrows, nose, eyes, and mouth contours; by detecting the key points, the basic contour of the face can be determined, and then the region of interest can be divided according to the key points.

[0009] In a possible implementation of the first aspect above, the key points include the facial contour and the locations of the facial features of the face in the face image, and the region of interest divided according to the key points includes the region obtained by removing the locations of the facial features from the face image.

[0010] In a possible implementation of the first aspect above, determining image features at a specific position in a facial image includes: determining a feature point in the facial image and taking the feature point as the specific position, wherein the image features of the feature point are different from image features of a surrounding area of ​​the feature point.

[0011] In a possible implementation of the first aspect above, the skin condition includes normal skin and problem skin, and the problem skin includes at least one type of problem skin.

[0012] In a possible implementation of the first aspect above, the type of problematic skin includes at least one of acne, mole or blackhead.

[0013] In a possible implementation of the first aspect above, the type corresponding to each problematic skin includes at least one level.

[0014] In a possible implementation of the first aspect, activating the enhanced content corresponding to the skin condition according to a preset operation for a specific position includes: determining that the operation for the display screen is a zoom operation for the specific position; determining to obtain the magnified parameter of the specific position in the face image; when the magnified parameter reaches a preset threshold, activating the enhanced content corresponding to the operation and the skin condition. This method can improve the fluency of user use and improve the user experience.

[0015] In a possible implementation of the first aspect, activating the enhanced content corresponding to the skin state according to a preset operation for a specific location includes: determining that the operation being performed is a preset click operation for the specific location; and activating the enhanced content corresponding to the operation and the skin state. The user can click on the skin he wants to view as needed, thereby improving the user's comfort.

[0016] In a possible implementation of the first aspect, activating the enhanced content corresponding to the skin state according to a preset operation for a specific location includes: determining that the operation is a first operation for the specific location; and activating the enhanced content corresponding to the operation and the skin state. This method can improve the user's usage fluency and improve the user experience.

[0017] In a possible implementation of the first aspect above, it further includes determining whether the enlarged parameter of a specific position in the face image is greater than or equal to a first threshold before activating the enhanced content corresponding to the operation and skin condition, and if so, activating the enhanced content.

[0018] In a possible implementation of the first aspect, the augmented content includes virtual content, or a combination of virtual content and real content.

[0019] In a possible implementation of the first aspect, the enhanced content is in the form of image, text, video, audio, or a combination of at least two of the forms.

[0020] In a possible implementation of the first aspect, the enhanced content includes at least one of an internal structure image of a subcutaneous tissue structure located below the dermis of a human face corresponding to a skin condition, a formation principle corresponding to the skin condition, and care suggestions for the skin condition. The user can take care of his or her skin according to the suggestions, thereby further improving the user experience.

[0021] In a second aspect, the present application provides a device for displaying skin details in an augmented reality manner, comprising:

[0022] An acquisition module, used for acquiring a face image to be analyzed;

[0023] A detection module for determining image features at specific locations in a face image;

[0024] The detection module obtains the skin condition corresponding to the image feature at a specific position;

[0025] The processing module is used to respond to a preset operation at a specific location, namely, to activate the enhanced content corresponding to the skin state, call the enhanced content corresponding to the skin state, and display the enhanced content.

[0026] According to the device for displaying skin details in an augmented reality manner according to the embodiment of the present application, it is possible to combine augmented content with real content (face image), magnify skin details, and display the internal structure of various skin problems, thereby enabling users to understand the root causes of skin problems, learn skin-related knowledge more deeply, and add interest.

[0027] In a possible implementation of the second aspect above, the detection module is specifically used to: determine the region of interest in the face image, determine the image features of a specific position in the region of interest, and by further determining the image features of the specific position within the region of interest, the accuracy of the image feature analysis can be improved.

[0028] In a possible implementation of the second aspect above, the detection module is specifically used to detect key points of the face in the face image, wherein the key points include: the contour of the face, the positions of the eyebrows, the nose, the eyes and the mouth contours; by detecting the key points, the basic contour of the face can be determined, and then the area of ​​interest can be divided according to the key points.

[0029] In a possible implementation of the second aspect above, the key points include the facial contour and the locations of the facial features of the face in the face image, and the region of interest divided according to the key points includes the region obtained by removing the locations of the facial features from the face image.

[0030] In a possible implementation of the second aspect above, the detection module is further used to: determine feature points in the face image and use the feature points as specific positions, wherein image features of the feature points are different from image features of surrounding areas of the feature points.

[0031] In a possible implementation of the second aspect, the skin condition includes normal skin and problem skin, and the problem skin includes at least one type of problem skin.

[0032] In a possible implementation of the second aspect, the type of problematic skin includes at least one of acne, mole or blackhead.

[0033] In a possible implementation of the second aspect, the type corresponding to each problematic skin includes at least one level.

[0034] In a possible implementation of the second aspect, the processing module is specifically used to: determine that the operation on the display screen is a zoom operation on a specific position; determine to obtain the zoomed-in parameter of the specific position in the face image; when the zoomed-in parameter reaches a preset threshold, activate the enhanced content corresponding to the operation and the skin condition. This method can improve the fluency of user use and improve the user experience.

[0035] In a possible implementation of the second aspect, the processing module is specifically used to: determine that the operation being performed is a preset click operation for a specific position; and activate enhanced content corresponding to the operation and the skin state. The user can click on the skin he wants to watch as needed, thereby improving the user's comfort.

[0036] In a possible implementation of the second aspect, the processing module is specifically used to: determine that the operation is a sliding operation on a specific position according to a preset trajectory; and activate enhanced content corresponding to the operation and skin state. This method can improve the user's fluency and improve the user experience.

[0037] In a possible implementation of the second aspect above, it further includes determining whether the magnified parameter of a specific position in the face image is greater than or equal to a first threshold before activating the enhanced content corresponding to the operation and skin condition, and if so, activating the enhanced content.

[0038] In a possible implementation of the second aspect, the augmented content includes virtual content, or a combination of virtual content and real content.

[0039] In a possible implementation of the second aspect, the enhanced content is in the form of one of images, text, video, and audio, or a combination of at least two of the forms.

[0040] In a possible implementation of the second aspect above, the enhanced content includes: an internal structure image of the subcutaneous tissue structure located below the dermis of the human face corresponding to the skin condition, a formation principle corresponding to the skin condition, and at least one of care recommendations for the skin condition.

[0041] In a third aspect, the present application provides an electronic device, including a collector and a processor connected to the collector, wherein the collector is used to obtain a face image, the processor is used to determine an image feature at a specific position in the face image, the processor obtains a skin state corresponding to the image feature at the specific position, and the processor is used to respond to a preset operation at the specific position, that is, to activate enhanced content corresponding to the skin state, and to call the enhanced content corresponding to the skin state so that the enhanced content is displayed on a display screen. By superimposing the skin state at the preset position and the enhanced content on the display screen, the user's interest in use is increased and the user experience is improved.

[0042] In a possible implementation of the third aspect above, the processor is specifically used to determine a region of interest in a facial image, and to determine image features at specific positions in the region of interest, which can improve the accuracy of image feature analysis.

[0043] In a possible implementation of the third aspect above, the processor is specifically used to detect key points of the face in the face image, where the key points include: the contour of the face, the positions of the eyebrows, the nose, the eyes and the mouth contours; the basic contour of the face can be determined by the processor through detection of the key points, and then the area of ​​interest can be divided according to the key points.

[0044] In a possible implementation of the third aspect above, the processor is further used to determine feature points in the face image and use the feature points as specific positions, wherein image features of the feature points are different from image features of surrounding areas of the feature points.

[0045] In a possible implementation of the third aspect, the processor is specifically used to determine that the operation on the display screen is a zoom operation on a specific position, and the processor determines to obtain the magnified parameter of the specific position in the face image. When the magnified parameter reaches a preset threshold, the processor activates the enhanced content corresponding to the operation and the skin condition. This method can improve the fluency of user use and improve the user experience.

[0046] In a possible implementation of the third aspect, the processor is specifically configured to determine that the operation being performed is a preset click operation for a specific position, so as to activate enhanced content corresponding to the operation and the skin state. The user can click the skin he wants to view as needed, thereby improving the user's comfort.

[0047] In a possible implementation of the third aspect, the processor is specifically configured to determine that the operation is a first operation for a specific position, and the processor activates enhanced content corresponding to the operation and the skin state. This method can improve the user's use fluency and improve the user experience.

[0048] In a possible implementation of the third aspect above, the processor is further used to determine whether the magnification of a specific position in the face image is greater than or equal to a first threshold before activating the enhanced content corresponding to the operation and skin condition, and if so, activate the enhanced content.

[0049] In a fourth aspect, the present application provides an electronic device, including a processor and a memory, wherein the memory stores instructions, and the processor is used to read the instructions stored in the memory to execute the method of the above-mentioned first aspect embodiment.

[0050] In a fifth aspect, the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the processor executes the method of the above-mentioned first aspect embodiment. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 A schematic diagram of an application scenario of augmented reality display of a face image according to an embodiment of the present application;

[0052] Figure 2 A schematic diagram of an internal structure model diagram corresponding to a skin state according to an embodiment of the present application;

[0053] Figure 3 This is a schematic diagram of a user operating a mobile phone interface according to an embodiment of the present application;

[0054] Figure 4 A flowchart of a method for skin detail magnification augmented reality display according to an embodiment of the present application;

[0055] Figure 5 This is a schematic diagram of the structure of key points of a face according to an embodiment of the present application;

[0056] Figure 6 A flowchart of a method for activating enhanced content corresponding to a skin state according to an embodiment of the present application;

[0057] Figure 7 A flowchart of a method for activating enhanced content corresponding to a skin state according to another embodiment of the present application;

[0058] Figure 8 A flowchart of a method for activating enhanced content corresponding to a skin state according to another embodiment of the present application;

[0059] Fig. 9 A scene diagram of a user operating a mobile phone interface according to an embodiment of the present application;

[0060] Fig.10 This is a flow chart of a user operating a mobile phone interface according to an embodiment of the present application;

[0061] Fig.11A device for displaying skin detail augmented reality according to an embodiment of the present application;

[0062] Fig.12 A schematic diagram of the structure of an electronic device according to an embodiment of the present application;

[0063] Fig.13 A block diagram of a device according to some embodiments of the present application;

[0064] Fig.14 A block diagram of a system on a chip (SoC) according to some embodiments of the present application. DETAILED DESCRIPTION

[0065] The embodiments of the present application will be further described below in conjunction with the accompanying drawings.

[0066] According to some embodiments of the present application, a method and a related device for displaying skin details in an augmented reality manner are disclosed. The method and device according to the present application can be applied to skin in various parts of the human body. In the specific implementation of the present application, for the sake of simplicity, the skin of the human face is used as an example for description.

[0067] Figure 1 FIG. 1 is a schematic diagram showing an application scenario of face image augmented reality display. Figure 1 As shown, user 101 collects facial images of himself or her or his friends through the camera of mobile terminal 102 so that the facial images are displayed on the display screen of mobile terminal 102. Parameters such as the resolution of the facial image mainly depend on the parameters of the camera component equipped with the mobile terminal 102, such as the resolution of the camera.

[0068] Furthermore, the detection module of the mobile terminal analyzes the facial image and obtains the skin condition of the user's facial image. The skin condition referred to here may include one of problem skin and normal skin (healthy skin) other than problem skin. For the skin on the face, problem skin includes but is not limited to one or more of blackheads, pimples and moles. The type (such as blackheads, pimples or moles, etc.), level (for example, divided into papules and pustules according to the severity of acne) and location information (such as location information of acne in the facial image) of the skin condition are stored.

[0069] Corresponding to the above skin types, different types of skin types, and different levels of each type of skin type, enhanced content is provided to display some additional information for the skin condition. These enhanced contents can be pre-stored in the local memory of the terminal device or in a remote server connected to the terminal device, so as to facilitate the rapid call of the terminal device. In addition, the enhanced content can also be the content obtained by the terminal device through machine learning based on the existing enhanced content, which is not limited here.

[0070] It is understandable that the augmented content of this application refers to virtual content that can be combined with the face image on the display screen of the terminal device through augmented reality (AR) technology. This application can improve the user's interest by combining the augmented content with the face image.

[0071] According to an embodiment of the present application, the augmented content may include virtual content in the form of one of images, text, video, and audio, or in a combination of at least two of the forms. For example, the augmented content may be presented in the form of a three-dimensional image of the internal structure model of the skin, showing an internal structure image of the subcutaneous tissue structure located below the dermis of the human face corresponding to the skin state.

[0072] In the above example, the internal structure model is displayed alone in the form of virtual content, or it can be displayed in combination with other real content or virtual content. For example, in addition to the internal structure model, care suggestions displayed in the form of video demonstrations, explanations of principles presented in audio, etc. are also displayed in combination.

[0073] The following description takes the enhanced content as the internal structure model as an example. Figure 2 The schematic diagram of the internal structure model diagram corresponding to the skin state is exemplified, such as Figure 2 As shown, skin conditions include acne, mole, blackhead and normal skin. Furthermore, acne can be divided into papules, pustules and the like. These skin conditions correspond to the acne internal structure model, the mole internal structure model, the blackhead internal structure model and the normal skin internal structure model, respectively. The acne internal structure model is further divided into the papule internal structure model and the pustule internal structure model.

[0074] According to one embodiment of the present application, Figure 3 This is a schematic diagram of a user operation mobile phone interface of an embodiment of the present application, such as Figure 3As shown, on the display screen 310, the user can zoom in on the face image 320 to more clearly observe the skin condition in the image. When the magnification of the image reaches a certain level, the display of enhanced content can be activated. Furthermore, when the user wants to understand the internal structure of the skin condition at a specific location in more detail, the user can zoom in, click on a specific location of the face image, or perform a first operation at a specific location in the face image, such as pressing the display screen with two fingers and gradually opening them to activate or trigger the internal structure model of the skin condition. For example, if the skin condition at the specific location is a blackhead, after the user continues to zoom in on the blackhead to a specified multiple, the internal structure model corresponding to the blackhead will be displayed in augmented reality on the display screen 310 of the mobile terminal, which can increase the fun of the user, improve the user experience, and meet the customer's exploration needs for the details of the skin condition.

[0075] The internal structure model of the skin state of the present application can obtain internal structure models corresponding to the skin states one by one through the existing model construction method, and store the internal structure model corresponding to the skin state in the mobile terminal device or the remote server.

[0076] It should be noted that the use of a mobile terminal in combination with a corresponding application to implement the method according to the present application is only an exemplary description, and the implementation of the present application is not limited to mobile terminals such as smart phones, but can also be other dedicated electronic devices with shooting and display functions, such as dedicated skin treatment devices, or electronic devices without shooting functions but that can receive and display images, or electronic devices without display functions but that can be connected to devices with display functions, which are not limited here. The enhanced content of the present application is universal and simple and easy to implement.

[0077] Based on the description of the above embodiments, according to some embodiments of the present application, the method for displaying skin details in augmented reality according to the embodiments of the present application is introduced with a specific embodiment below. Figure 1 The description is given by taking the smartphone as an example of a mobile terminal as an example. Figure 4 A flowchart of a method for enlarging skin detail augmented reality display according to the present application is shown, Figure 4 As shown, the method specifically includes:

[0078] Step S310, obtaining a face image to be analyzed, and displaying the face image on a display screen, wherein the face image can be obtained through an image acquisition function and a shooting function of a mobile terminal, and the obtained face image is displayed on a display screen of a smart phone. In other embodiments of the present application, the mobile terminal can also obtain images of other parts of the user's body in addition to the face image, such as hands, back, etc., which are not limited here.

[0079] Step S320, determining the image features of a specific position in the face image. The specific position may be a designated position manually selected by the user on the face image, such as the tip of the nose, cheek or forehead. The specific position may be obtained by the user freely clicking on the screen according to his or her own interests. For example, when the user wants to know the image features of a specific position in the face image, the user may click on the position corresponding to the specific position of the face image on the display screen of the terminal device to obtain the specific position in the face image, and further detect the specific position to obtain the image features of the specific position. The image features may be color features, pattern outlines, color density or a combination of the two or more features of the specific position. In other words, by mapping the coordinates of the position selected on the display screen to the specific position of the face image in proportion, and then by determining the image features at the specific position, the user can determine the image features of the preset position of the face image by clicking on the screen.

[0080] According to another embodiment of the present application, the image features at a specific position in a facial image can also be determined by automatically identifying and determining the specific position in the facial image through a terminal device based on a neural network facial recognition technology. That is, the face detection step using the facial recognition technology first determines the feature points in the facial image related to the skin condition. For example, when the image features of the feature points at a certain position in the facial image are identified to be different from the image features of the surrounding area, it can be determined that the certain position in the facial image is a specific position. This implementation method is more used to identify problematic skin.

[0081] As another implementation method, the user can use the naked eye to identify the difference between the color of the feature point and the color of the surrounding image. If the color of the feature point is different from the color of the surrounding image, the position is judged to be a feature point, and the feature point is used as a specific position. For example, when there is acne on a person's face, the red color of the acne is different from the yellow color of the surrounding skin. The user can determine the position of the acne on the image as a feature point to be processed through color difference analysis. In order to prevent the judgment of color from being affected by factors such as light and shooting angle, the user can choose a position and angle with suitable light to shoot the image so that the human eye can recognize the difference in color, which is convenient for the user to obtain a more realistic image.

[0082] According to one embodiment of the present application, before determining the image features of a specific position in a face image, a region of interest (ROI) in the face image can be determined first, so as to locate the position of interest more quickly and reduce the amount of calculation. The ROI area can be set to different areas according to different skin conditions. For example, blackheads usually appear on the nose, so the ROI area corresponding to the blackhead can be set at the tip of the nose. After the ROI area is determined, the specific position is further determined, and it is judged in which ROI area the specific position is located, which is conducive to determining the image features of the specific position within the range of the RIO area. For example, the specific position is a certain position of the tip of the nose. Since the position of the tip of the nose is the ROI area corresponding to the blackhead, it is preliminarily judged that the skin condition of the image corresponding to the specific position may be a blackhead. Through the judgment result, the image features are further extracted, which can improve the accuracy of the image feature analysis.

[0083] The division of the ROI area is described in detail below with reference to the accompanying drawings. Figure 5 Figure 2 shows a schematic diagram of the structure of facial key points. Figure 5 As shown in the figure, the key points of the face in the face image are detected, and the ROI region of interest is divided by the key points. Specifically, the key points include: the contour of the face, the location of the eyebrows, the nose, the eyes and the mouth contour. Different key point detection algorithms set different numbers of key points, such as Figure 5 As shown, 68 key points can be set, such as Figure 5 As shown, there are 68 key points from 0 to 67, and a denser number of 98, 1000, 4000 or more key points can also be set, which is not limited here. The basic contour of the face can be determined by detecting the key points, and then the ROI region of interest can be divided according to the key points, such as the upper part of the eyebrows can be divided into the forehead 501, the lower part of the eyes and the two sides of the nose can be divided into the cheek 502, and the lower part of the mouth can be divided into the jaw 503 and other ROI regions. Considering that some parts of the face are not part of the skin, such as lips, eyes and nostrils, the division of the ROI region of interest can bypass these parts, that is, the ROI region of interest can be the facial skin without lips, eyes and nostrils.

[0084] According to one embodiment of the present application, determining the image features of a specific position in a face image includes detecting the skin of a region of interest (ROI) or a specific position, and determining feature points in the face image, wherein the feature points are used as specific positions. The image features of the feature points are different from the image features of the surrounding areas of the feature points. For example, the color features, pattern outlines, color density, or a combination of the two or more features of the specific position are different. Automatic recognition can be achieved and the specific position can be obtained through the detection of the feature points.

[0085] The following takes color features as an example to describe the image features at specific locations. For example, the skin details of a face image are detected through a skin detection module, wherein the skin detection method can adopt a conventional detection method. For example, for the detection of acne, the input color RGB (Red, Green, Blue) image can be converted into a grayscale image, and the maximum value of the grayscale in each area is found. The grayscale image is normalized using the maximum value obtained, and the color RGB image is converted into the HSV (Hue, Saturation, Value) color space, and the V channel in the HSV color space is extracted and normalized. The normalized grayscale image is subtracted from the normalized V channel to obtain the characteristic points of the acne.

[0086] Those skilled in the art can also understand that, after a feature point is confirmed by the analysis method, the ROI area can also be confirmed based on the feature point, for example, based on the identified feature point, a circle is drawn with a predetermined radius, and the circular area is the ROI area determined based on the feature point. Other possible feature points can be further confirmed in the ROI area.

[0087] Step S330, obtaining the skin condition corresponding to the image feature at the specific position. The skin condition may include the above Figure 2 The skin condition shown is, for example, problematic skin with blackheads, pimples, and moles. In other embodiments of the present application, other skin conditions may be present, such as skin conditions with spots, fine lines, etc., which are not limited here. For example, a user clicks on the location of a pimple on the nose of a person's face on the display screen of a terminal device, and the terminal device can determine through the detection module that the skin condition corresponding to the location is a pimple.

[0088] Step S340, according to the specified operation for the specific position, the enhanced content corresponding to the skin state is activated, and the enhanced content is called to enhance the display of the enhanced content and the face image. The enhanced content is pre-stored, including virtual content, or a combination of virtual content and real content. By superimposing the skin state of the preset position and the enhanced content on the display screen, the user's interest is improved and the user experience is enhanced.

[0089] According to one embodiment of the present application, the enhanced content may include content in the form of one of images, text, video, and audio, or in a combination of at least two of the forms, such as an internal structure image of the subcutaneous tissue structure located below the dermis of the human face corresponding to the skin condition, a formation principle corresponding to the skin condition, and at least one of care suggestions for the skin condition. For example, the enhanced content is an internal structure image corresponding to the skin condition. When the internal structure image is activated through an operation, the internal structure image is superimposed on the skin condition at a specific location and displayed on the display screen, so that the user can more clearly understand the internal structure of the skin condition at a specific location, meet the user's need for further exploration of the skin condition, and enhance the user's experience.

[0090] According to an embodiment of the present application, activating enhanced content corresponding to the skin state through a designated operation at a specific location may include the following exemplary implementations.

[0091] The first implementation method is Figure 6 As shown, the following steps are included:

[0092] Step S510, determining that the operation on the screen of the display screen is a zoom operation on a specific position, wherein the zoom operation on a specific position (such as a pimple, blackhead or mole) refers to a gesture operation by touching the display screen with a finger, for example, pressing the display screen with two fingers and gradually opening them, etc. Before and after the operation, the specific position is magnified several times on the display screen, and when the pixel size of the specific position after the zoom operation is greater than the pixel size of the specific position before the zoom operation, it is determined that the zoom operation on the specific position is being implemented. As an alternative operation mode, the user can select the magnification of the specific position by selecting the magnification option.

[0093] Step S520, determining the magnification of a specific position in the acquired face image, by detecting the magnification of a specific position (such as a pimple, a blackhead, or a mole), for example, the position where the blackhead is located is magnified N times before and after the user's operation, where N is a natural number greater than or equal to 1. For example, if the image is magnified 3 times, the magnification of the specific position is 3 times.

[0094] Step S530, when the magnification parameter reaches a preset threshold, for example, the first threshold is that the specific position is magnified 5 times before and after the user's operation, when the threshold of the specific position being magnified 5 times is met, that is, once the image containing the feature point is magnified more than 5 times, the enhanced content corresponding to the operation and skin condition is activated, such as the internal structure model of acne.

[0095] According to another embodiment of the present application, Figure 7 As shown, the second implementation method of activating the enhanced content corresponding to the skin state by operating at a specific position may include the following steps:

[0096] Step S510, determining whether the operation being performed is a preset point selection operation for a specific location, wherein the point selection operation can be an operation according to preset rules such as a single click, double click or multiple clicks of a mouse directly on the display screen by a user's finger, or an operation such as a long press on a specific location.

[0097] Step S520, when the user clicks a specific location according to a predetermined rule (single click, double click, or multiple clicks, etc.), enhanced content corresponding to the user's operation and skin condition will be activated, such as an acne internal structure model.

[0098] When the user clicks or double-clicks a specific location to activate the display of enhanced content, the selection operation will be performed on the basis of the enlarged image for the sake of accuracy. In combination with the first embodiment, the magnification ratio can be combined as one of the conditions for triggering enhanced content. For example, when the image including the feature point is magnified to at least 5 times, the display of enhanced content such as the internal structure model of the acne is activated by clicking on the acne part. On the contrary, when the image is only magnified to 3 times, selecting the feature point will not activate the enhanced content.

[0099] According to another embodiment of the present application, Figure 8 As shown, the third way of activating the enhanced content corresponding to the skin state by the first operation at a specific position may include the following steps:

[0100] Step S710, determining whether the operation on the feature point is a sliding operation on a specific position along a preset trajectory, wherein the sliding operation may be sliding along a preset trajectory around the specific position, such as sliding an arc, a circle, or a straight line.

[0101] Step S720, when it is detected that the operation being implemented slides along a preset trajectory, such as drawing a circle around a specific point, or it can be a special on-screen operation pre-customized by the user, such as drawing a specific letter trajectory, the enhanced content corresponding to the skin condition of the feature point is activated.

[0102] When the user slides at a specific location, or slides on the screen with a specially defined trajectory to activate the display of enhanced content, the sliding operation will be performed on the basis of the enlarged image for the sake of accuracy of the operation. In combination with the first embodiment above, the magnification ratio can be combined as one of the conditions for triggering enhanced content. For example, when the image including the feature point is magnified to at least 5 times, the display of enhanced content such as the internal structure model of the acne is activated by drawing a circle around part of the acne. On the contrary, when the image is only magnified to 3 times, drawing a circle around the feature point will not activate the enhanced content.

[0103] In some other embodiments of the present application, the operation may be a combination of clicking and sliding operations to activate enhanced content corresponding to the operation and skin condition, which is not limited here.

[0104] The following describes the method of displaying skin details of the present application by taking a mobile phone as an example and combining it with a specific usage scenario. Fig. 9 The following diagram shows a scene diagram of a user operating a mobile terminal interface, such as Fig. 9 As shown, the user first opens the corresponding App application and takes a face image, and then the basic skin condition of the face image is detected by the face detection module of the mobile terminal, and the basic skin condition of the user is comprehensively displayed, such as the number and location information of pimples, blackheads, and moles, as well as the comprehensive score of the user's skin condition. When the user wants to further understand the more detailed structure of a certain skin condition, he or she can select the skin condition of interest.

[0105] The following describes the method of displaying skin details using acne as an example. Fig.10 A flowchart of a user operating a mobile terminal interface is shown. Fig.10 As shown in a in FIG. 1 , the user first clicks on a face image to display the face image on the display screen of the mobile terminal. Fig.10 As shown in b and c in FIG, the user slides two fingers on the display screen to zoom in on the face image and detect the magnification factor, which makes it easier for the user to observe the skin details. Fig.10 As shown in c in FIG. 1 , the user selects a specific position with a finger, and the selected position is a pustule. After selecting, the user slides the finger upward to activate the pustule internal structure model image corresponding to the pustule, so that the pustule internal structure model and the corresponding pustule are displayed in augmented reality, as shown in FIG. Fig.10 As shown in d in . Further, Fig.10 As shown in e in the figure, by clicking on a specific position or indication, such as a specific symbol, etc., you can further understand the formation principle of the skin condition or the text or audio information such as the care suggestions for the skin condition. When the user wants to return to the previous level, he can click the return to the previous level button or perform gesture operations, such as pressing two fingers on the display screen and pinching. It is understandable that the image can also be reduced by pinching with two fingers.

[0106] The method of displaying skin details in an augmented reality manner in the embodiment of the present application can combine virtual images with real skin images and display them to users in an augmented reality manner, thereby satisfying users' exploration of the root causes of skin problems and their needs for learning skin knowledge. Users can not only view and manage their facial skin conditions in real time, but also gain a more intuitive and interesting experience through augmented reality display. Users can have a deeper and more in-depth understanding of their skin problems, learn skin-related knowledge, and obtain skin care advice, which is both entertaining and educational and improves the user experience.

[0107] According to some embodiments of the present application, a device for displaying skin detail augmented reality is disclosed. Fig.11 FIG. 4 shows a schematic diagram of the structure of a skin detail augmented reality display device. Fig.11 As shown, the display device includes:

[0108] Display screen 1001;

[0109] In acquisition module 1002, the user obtains the facial image to be analyzed and displays the facial image on a display screen. In other embodiments of the application, the facial image may also be an image of other parts of the user's body, which is not the only limitation herein.

[0110] The detection module 1003 is used to determine the image features of a specific position in the face image, such as the color features, pattern outlines, color density or a combination of the two or more at the specific position.

[0111] The detection module 1003 is used to obtain the skin state corresponding to the image feature at a specific position, wherein the skin state includes normal skin and problem skin, the problem skin includes at least one type of problem skin, the type of problem skin includes at least one of acne, mole or blackhead, and the type of each problem skin includes at least one level, such as acne is divided into two levels of papules and pustules according to the severity. In other embodiments of the present application, other skin states may be used, such as spots, fine lines and other skin states, which are not limited here.

[0112] The processing module 1004 activates the augmented content corresponding to the skin state according to the preset operation for the specific position, and displays the augmented content on the display screen by superimposing the face image. The augmented content includes virtual content, or a combination of virtual content and real content. The augmented content can be pre-stored in the terminal or temporarily called from a remote server.

[0113] According to an embodiment of the present application, the enhanced content exists in the form of one of images, text, video, and audio, or in a combination of at least two of the forms, such as at least one of an internal structure image of a subcutaneous tissue structure located below the dermis of a human face corresponding to a skin condition, a formation principle corresponding to a skin condition, and nursing suggestions for the skin condition.

[0114] According to an embodiment of the present application, the detection module 1003 is also used to determine an ROI region of interest in the face image, and determine the image features of a specific position in the ROI region of interest. The ROI region can be set to different ROI regions of interest according to different skin conditions. For example, blackheads usually appear on the nose, so the ROI region corresponding to the blackhead can be set at the tip of the nose, and the image features of a specific position are determined in the ROI region to improve the accuracy of the image feature analysis.

[0115] Furthermore, the detection module 1003 is also specifically used to detect key points of the face of the human face in the face image, and divide the ROI region of interest according to the key points. The key points include the facial contour and the location of the facial features of the human face in the face image, and the ROI region of interest divided according to the key points includes the area obtained by removing the location of the facial features from the face image.

[0116] According to an embodiment of the present application, the detection module 1003 is further used to: determine feature points in the face image, and use the feature points as specific positions, wherein image features of the feature points are different from image features of surrounding areas of the feature points.

[0117] According to one embodiment of the present application, the processing module 1004 is specifically used to: determine whether the operation is a zoom operation on a specific position, determine to obtain the magnified parameters of the specific position in the face image, and when the magnified parameters reach a first threshold, activate the enhanced content corresponding to the operation and skin condition.

[0118] According to one embodiment of the present application, the processing module 1004 is specifically used to determine whether the operation is a preset point selection operation for a specific location, and activate the enhanced content corresponding to the operation and the skin state, wherein the point selection operation includes the user directly using a finger on the display screen or using a mouse to single-click, double-click or multi-click according to preset rules, or long press on a specific location.

[0119] According to one embodiment of the present application, the processing module 1004 is specifically used to determine whether the operation is a sliding operation along a preset trajectory at a specific location, and activate enhanced content corresponding to the operation and skin state, wherein the sliding operation includes sliding along a preset trajectory around a specific location, such as sliding an arc, a circle, or a straight line.

[0120] In the present application, the functions and working processes of the various components of the display device have been described in detail in the above embodiments. For details, please refer to the method of displaying skin details in an augmented reality manner in the above embodiments, which will not be repeated here.

[0121] According to the device for skin detail augmented reality display according to the embodiment of the present application, it is possible to combine virtual images with real skin images and present them to users in an augmented reality manner, thereby combining education with entertainment and improving the user experience.

[0122] According to some embodiments of the present application, an electronic device is disclosed. Fig.12 Schematic diagram of the structure of the electronic device is shown. Fig.12 As shown, specifically, the electronic device may include:

[0123] The collector 1103 is used to collect the user's face image.

[0124] Memory 1102, used for storing instructions;

[0125] A processor 1104 is configured to read instructions stored in the memory to execute the method for displaying skin details in an augmented reality manner according to the above embodiment; and

[0126] The display device is implemented as a display screen, and is used for augmented reality display of augmented content and real skin condition images.

[0127] Although in the above embodiments, the electronic device itself includes a display screen for displaying, for example, enhanced content and real skin condition images, those skilled in the art will appreciate that some electronic devices may not have a display screen themselves. The electronic device according to the embodiments of the present application can synchronize data to a terminal device with a display screen such as a smart phone, tablet computer, desktop PC, and notebook PC that is communicatively connected to the electronic device, so as to display the corresponding augmented reality display image to the user.

[0128] According to the electronic device of the embodiment of the present application, it is possible to combine a virtual image with a real skin image and present it to the user in an augmented reality manner, thereby combining education with entertainment and improving the user experience.

[0129] Reference now Fig.13, shown is a block diagram of a device 1200 according to one embodiment of the present application. The device 1200 may include one or more processors 1201 coupled to a controller hub 1203. For at least one embodiment, the controller hub 1203 communicates with the processor 1201 via a multi-drop bus such as a front side bus (FSB), a point-to-point interface such as a Quick Path Interconnect (QPI), or a similar connection 1206. The processor 1201 executes instructions that control general types of data processing operations. In one embodiment, the controller hub 1203 includes, but is not limited to, a graphics memory controller hub (GMCH) (not shown) and an input / output hub (IOH) (which may be on a separate chip) (not shown), wherein the GMCH includes a memory and a graphics controller and is coupled to the IOH.

[0130] The device 1200 may also include a coprocessor 1202 and a memory 1204 coupled to the controller hub 1203. Alternatively, one or both of the memory and the GMCH may be integrated within the processor (as described in the present application), with the memory 1204 and the coprocessor 1202 directly coupled to the processor 1201 and the controller hub 1203, with the controller hub 1203 and the IOH being in a single chip. The memory 1204 may be, for example, a dynamic random access memory (DRAM), a phase change memory (PCM), or a combination of the two. In one embodiment, the coprocessor 1202 is a special-purpose processor, such as, for example, a high-throughput MIC processor (Many Integerated Core, MIC), a network or communication processor, a compression engine, a graphics processor, a general-purpose graphics processor (General Purpose Computing on GPU, GPGPU), or an embedded processor, etc. The optional properties of the coprocessor 1202 are indicated by dashed lines in Fig.13 middle.

[0131] The memory 1204, as a computer-readable storage medium, may include one or more tangible, non-transitory computer-readable media for storing data and / or instructions. For example, the memory 1204 may include any suitable non-volatile memory such as flash memory and / or any suitable non-volatile storage device, such as one or more hard disk drives (HDD(s)), one or more compact disc (CD) drives, and / or one or more digital versatile disc (DVD) drives.

[0132] In one embodiment, the device 1200 may further include a network interface (Network Interface Controller, NIC) 1206. The network interface 1206 may include a transceiver for providing a radio interface for the device 1200 to communicate with any other suitable device (such as a front-end module, an antenna, etc.). In various embodiments, the network interface 1206 may be integrated with other components of the device 1200. The network interface 1206 may implement the functions of the communication unit in the above-mentioned embodiments.

[0133] The device 1200 may further include input / output (I / O) devices 1205. The I / O 1205 may include: a user interface designed to enable a user to interact with the device 1200; a peripheral component interface designed to enable peripheral components to interact with the device 1200; and / or sensors designed to determine environmental conditions and / or location information related to the device 1200.

[0134] It is worth noting that Fig.13 This is for illustrative purposes only. Fig.13 It is shown that the device 1200 includes multiple components such as a processor 1201, a controller hub 1203, and a memory 1204. However, in actual applications, the device using the methods of the present application may only include a part of the components of the device 1200, for example, it may only include the processor 1201 and the NIC 1206. Fig.13 The properties of optional devices are shown with dotted lines.

[0135] According to some embodiments of the present application, instructions are stored on the memory 1204 as a computer-readable storage medium. When the instructions are executed on the computer, the system 1200 executes the method according to the above embodiment. For details, please refer to the method of the above embodiment, which will not be repeated here.

[0136] Reference now Fig.14, which is a block diagram of a SoC (System on Chip) 1300 according to an embodiment of the present application. Fig.14 In the , similar components have the same reference numerals. In addition, the dashed boxes are optional features of more advanced SoCs. Fig.14 In the embodiment, SoC 1300 includes: an interconnect unit 1350, which is coupled to an application processor 1310; a system agent unit 1380; a bus controller unit 1390; an integrated memory controller unit 1340; a group or one or more coprocessors 1320, which may include integrated graphics logic, an image processor, an audio processor, and a video processor; a static random access memory (SRAM) unit 1330; and a direct memory access (DMA) unit 1360. In one embodiment, the coprocessor 1320 includes a dedicated processor, such as, for example, a network or communication processor, a compression engine, a GPGPU, a high throughput MIC processor, or an embedded processor.

[0137] The static random access memory (SRAM) unit 1330 may include one or more computer-readable media for storing data and / or instructions. The computer-readable storage medium may store instructions, specifically, temporary and permanent copies of the instructions. The instruction may include: when executed by at least one unit in the processor, the Soc 1300 executes the calculation method according to the above embodiment, which may be specifically referred to the method of the above embodiment, and will not be repeated here.

[0138] The various embodiments of the mechanism disclosed in the present application can be implemented in hardware, software, firmware or a combination of these implementation methods. The embodiments of the present application can be implemented as a computer program or program code executed on a programmable system, which includes at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device and at least one output device.

[0139] Program code can be applied to input instructions to perform the functions described in this application and generate output information. The output information can be applied to one or more output devices in a known manner. For the purposes of this application, a processing system includes any system having a processor such as, for example, a digital signal processor (DSP), a microcontroller, an application specific integrated circuit (ASIC), or a microprocessor.

[0140] Program code can be implemented with high-level programming language or object-oriented programming language to communicate with the processing system. When necessary, program code can also be implemented with assembly language or machine language. In fact, the mechanism described in this application is not limited to the scope of any specific programming language. In either case, the language can be a compiled language or an interpreted language.

[0141] In some cases, the disclosed embodiments may be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions carried or stored on one or more temporary or non-temporary machine-readable (e.g., computer-readable) storage media, which may be read and executed by one or more processors. For example, instructions may be distributed over a network or through other computer-readable media. Therefore, machine-readable media may include any mechanism for storing or transmitting information in a machine (e.g., computer) readable form, including, but not limited to, floppy disks, optical disks, optical disks, compact disc read-only memories (Compact Disc Read Only Memory, CD-ROMs), magneto-optical disks, read-only memories (ReadOnly Memory, ROM), random access memories (RAM), erasable programmable read-only memories (ErasableProgrammable Read Only Memory, EPROM), electrically erasable programmable read-only memories (ElectricallyErasableProgrammableReadOnlyMemory, EEPROM), magnetic cards or optical cards, flash memory, or tangible machine-readable memories for transmitting information (e.g., carrier waves, infrared signal digital signals, etc.) using the Internet in electrical, optical, acoustic, or other forms of propagation signals. Accordingly, machine-readable media include any type of machine-readable media suitable for storing or transmitting electronic instructions or information in a form readable by a machine (eg, a computer).

[0142] In the accompanying drawings, some structural or method features may be shown in a specific arrangement and / or order. However, it should be understood that such a specific arrangement and / or order may not be required. Instead, in some embodiments, these features may be arranged in a manner and / or order different from that shown in the accompanying drawings of the specification. In addition, the inclusion of structural or method features in a particular figure does not mean that such features are required in all embodiments, and in some embodiments, these features may not be included or may be combined with other features.

[0143] It should be noted that the units / modules mentioned in the various device embodiments of the present application are all logical units / modules. Physically, a logical unit / module can be a physical unit / module, or a part of a physical unit / module, or can be implemented as a combination of multiple physical units / modules. The physical implementation method of these logical units / modules themselves is not the most important. The combination of functions implemented by these logical units / modules is the key to solving the technical problems proposed by the present application. In addition, in order to highlight the innovative part of the present application, the above-mentioned device embodiments of the present application do not introduce units / modules that are not closely related to solving the technical problems proposed by the present application, which does not mean that there are no other units / modules in the above-mentioned device embodiments.

[0144] It should be noted that in the examples and description of this patent, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "including one" do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0145] Although the present application has been illustrated and described with reference to certain preferred embodiments thereof, it will be apparent to those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the present application.

Claims

1. A method for displaying skin details, characterized in that: include: The electronic device acquires a facial image; The electronic device determines an image feature at a specific position in the face image; The electronic device acquires a skin condition corresponding to the image feature at the specific position; The electronic device responds to the preset operation of the specific position and displays the enhanced content corresponding to the skin state, wherein the enhanced content is pre-stored and includes virtual content, or a combination of virtual content and real content.

2. The method according to claim 1, characterized in that The electronic device determines the image feature at a specific position in the face image, including: The electronic device determines a region of interest in the face image, and determines image features at a specific position in the region of interest.

3. The method according to claim 2, characterized in that The determining of the region of interest on the face image comprises: Detecting key points of the face in the face image; A region of interest is divided according to the key points.

4. The method according to claim 3, characterized in that The key points include the facial contour and the location of the facial features of the face in the face image, and The region of interest divided according to the key points includes a region obtained by removing the positions of the facial features from the face image.

5. The method according to claim 1 or 2, characterized in that: The electronic device determines the image feature at a specific position in the face image, including: Determine a feature point in the face image and use the feature point as the specific position, The image features of the feature points are different from the image features of the surrounding areas of the feature points.

6. The method according to claim 1, characterized in that The electronic device activates the enhanced content corresponding to the skin state according to a preset operation for the specific position, including: Determining that the operation on the display screen is a zoom-in operation on the specific position; Determine and obtain the magnified parameter of a specific position in the face image; When the amplified parameter reaches a preset threshold, the enhanced content corresponding to the operation and the skin condition is activated.

7. The method according to claim 1, characterized in that The electronic device activates the enhanced content corresponding to the skin state according to a preset operation for the specific position, including: Determining that the operation being performed is a preset point selection operation for the specific location; Activate enhanced content corresponding to the operation and the skin state.

8. The method according to claim 1, characterized in that The electronic device activates the enhanced content corresponding to the skin state according to a preset operation for the specific position, including: Determining that the operation is a first operation for the specific position; Activate enhanced content corresponding to the operation and the skin state.

9. The method according to claim 7 or 8, characterized in that: Further including: Before activating the enhanced content corresponding to the operation and the skin condition, it is determined whether the magnification of the specific position in the face image is greater than or equal to a first threshold, and if so, the enhanced content is activated.

10. The method according to claim 1, characterized in that The enhanced content exists in the form of one of images, text, video, and audio, or a combination of at least two of the forms.

11. The method according to claim 10, characterized in that The enhanced content includes: at least one of an internal structure image of a subcutaneous tissue structure located below the dermis of a human face corresponding to the skin condition, a formation principle corresponding to the skin condition, and care suggestions for the skin condition.

12. An electronic device, characterized in that: include: A collector, used to obtain face images; a processor for determining image features at specific locations in the facial image; The processor acquires a skin condition corresponding to an image feature at a specific position; The processor is used to respond to a preset operation at a specific location, namely, activating the enhanced content corresponding to the skin state, calling the enhanced content corresponding to the skin state so that the enhanced content is displayed on the display screen, and the enhanced content is pre-stored and includes virtual content, or a combination of virtual content and real content.

13. The device according to claim 12, characterized in that The processor is specifically used to: determine a region of interest in the face image, and determine image features at a specific position in the region of interest.

14. The device according to claim 13, characterized in that The processor is further specifically configured to: Detecting key points of the face in the face image; A region of interest is divided according to the key points.

15. The device according to claim 12 or 13, characterized in that The processor is further configured to: Determine a feature point in the face image and use the feature point as the specific position, The image features of the feature points are different from the image features of the surrounding areas of the feature points.

16. The device according to claim 12, characterized in that The processor is specifically used for: Determining that the operation on the display screen is a zoom-in operation on the specific position; Determine and obtain the magnified parameter of a specific position in the face image; When the amplified parameter reaches a preset threshold, the enhanced content corresponding to the operation and the skin condition is activated.

17. The device according to claim 12, characterized in that The processor is specifically used for: Determining that the operation being performed is a preset point selection operation for the specific location; Activate enhanced content corresponding to the operation and the skin state.

18. The device according to claim 12, characterized in that The processor is specifically used for: Determining that the operation is a first operation for the specific position; Activate enhanced content corresponding to the operation and the skin state.

19. The device according to claim 17 or 18, characterized in that The processor is further configured to: Before activating the enhanced content corresponding to the operation and the skin condition, it is determined whether the magnification of the specific position in the face image is greater than or equal to a first threshold, and if so, the enhanced content is activated.

20. An electronic device, characterized in that: including a processor and memory, The memory stores instructions. The processor is used to read the instruction stored in the memory to execute the method according to any one of claims 1 to 11.

21. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor executes the method according to any one of claims 1 to 11.

Images