AR-based skin detection method, device and system

Abstract

The invention discloses a skin detection method, device and system based on AR (Augmented Reality). The method comprises the following steps: acquiring a full face image; performing skin detection on the full face image, obtaining a problem skin type and a region where the problem skin is located, taking the region where the problem skin is located as a recommendation region, and performing enhanced display on the recommendation region; when it is captured that the local image acquisition device makes contact with the recommendation area, obtaining a problem skin type corresponding to the recommendation area; based on the acquired problem skin type, controlling local image acquisition equipment to acquire a local image under a specified light source; and receiving the local image sent by a local image acquisition device, and detecting the problem skin based on the local image. According to the invention, the problem skin type is detected based on the full face image, the recommendation area is enhanced and displayed on the full face image, the local image acquisition device is guided to quickly align to the problem skin, and the local image acquisition device is controlled to turn on different light sources based on different problem skin types to detect the problem skin more carefully and more accurately.

Description

Technical Field

[0001] The present invention relates to the technical field of skin detection, and in particular to an AR-based skin detection method, device and system. Background Art

[0002] With the continuous development of science and technology, people's demand for beauty is also increasing. As the largest organ of the human body, the health of the skin is crucial to beauty. Traditional skin testing methods usually rely on professional dermatologists and instruments. This traditional skin testing method has many defects, such as taking a long time and high costs.

[0003] There are currently two detection methods on the market:

[0004] (1) Using the front camera of a mobile phone or other electronic device to directly collect facial photos for multiple types (dimensions) of skin detection (such as skin color, spots, acne, wrinkles, moisture content, oil secretion, etc.);

[0005] (ii) Use a device specifically used for collecting skin images (such as a skin test pen) to collect local facial images, and then perform multi-dimensional detection of the local skin.

[0006] Both current solutions have drawbacks. The first detection method can only roughly detect the more obvious skin problem dimensions on the face, such as spots, acne, etc., and cannot accurately detect more detailed dimensions such as pores, oil in the pores, and blackheads. Although the second detection method can detect facial skin in a relatively detailed manner, because it is local imaging and the area detected each time is relatively small, when detecting facial skin, if you want to detect every position on the entire face, you need to repeat the operation multiple times. The detection lacks specificity and cannot guide users to detect skin problem areas in a targeted manner. Summary of the invention

[0007] The main purpose of the present invention is to propose an AR-based skin detection method, device and system, which overcome the shortcomings of the prior art, detect the problem skin type based on the full face image, enhance the display of the recommended area on the full face image, and then guide the local image acquisition equipment to quickly align with the problem skin, and turn on different light sources based on different problem skin types to perform more detailed, accurate and faster detection of the problem skin.

[0008] The present invention adopts the following technical solution:

[0009] In one aspect, a skin detection method based on AR includes:

[0010] Get full face image;

[0011] Perform skin detection on the full-face image to obtain the problematic skin type and the area where it is located, use the area where it is located as a recommended area, and enhance the display of the recommended area;

[0012] When it is captured that the local image acquisition device contacts the recommended area, the problematic skin type corresponding to the recommended area is obtained;

[0013] Based on the acquired problematic skin type, controlling the local image acquisition device to acquire the local image under the specified light source;

[0014] The local image sent by the local image acquisition device is received, and the problematic skin is detected based on the local image.

[0015] Preferably, before capturing that a local image acquisition device has touched the recommended area, the method further includes:

[0016] The current acquisition area of ​​the local image acquisition device is acquired in real time, and the current acquisition area is enhanced and displayed.

[0017] Preferably, when the local image acquisition device is captured to contact the recommended area, obtaining the problematic skin type corresponding to the recommended area specifically includes:

[0018] When it is captured that the current acquisition area of ​​the local image acquisition device is located in the recommended area, the problematic skin type corresponding to the recommended area is obtained.

[0019] Preferably, the method for acquiring the current acquisition area of ​​the local image acquisition device includes:

[0020] Acquiring a facial image including a local image acquisition device;

[0021] Identify the coordinate position of the front-end preset component of the local image acquisition device in the face image;

[0022] The area where the coordinate position is located is used as the current collection area.

[0023] Preferably, the method for acquiring the current acquisition area of ​​the local image acquisition device includes:

[0024] Acquiring a facial image including a handheld local image acquisition device;

[0025] Identify the coordinate position of a certain hand key point in the face image, or identify the coordinate position of the center of two or more hand key points in the face image;

[0026] The area where the coordinate position is located is used as the current collection area.

[0027] Preferably, the method for acquiring the current acquisition area of ​​the local image acquisition device includes:

[0028] Acquiring a facial image including a handheld local image acquisition device;

[0029] Identify the coordinate position of the front-end preset component of the local image acquisition device in the face image, and identify the coordinate position of a key point of a hand in the face image;

[0030] Calculate the coordinate position of the center of the coordinate position of the front-end preset component and the coordinate position of a certain hand key point;

[0031] The area where the center coordinates are located is taken as the current acquisition area.

[0032] Preferably, the method for acquiring the current acquisition area of ​​the local image acquisition device includes:

[0033] Acquiring a facial image including a handheld local image acquisition device;

[0034] Identify the coordinate positions of the front-end preset components of the local image acquisition device in the face image, and identify the coordinate positions of two or more key points of the hand in the face image;

[0035] Calculate the coordinate position of the front-end preset component and the coordinate position of the center of the coordinate positions of two or more hand key points;

[0036] The area where the center coordinates are located is taken as the current acquisition area.

[0037] Preferably, the method for defining the area where the coordinate position is located includes:

[0038] Based on the acquired face image, the face key point detection method is used to divide the face into preset areas according to the face key points;

[0039] Get the contour coordinates of each region;

[0040] Determine the contour to which the coordinate position belongs;

[0041] Use the corresponding contour as the current acquisition area.

[0042] Preferably, the problematic skin type and the area where it is located are obtained, and the area where it is located is used as the recommended area, specifically including:

[0043] The problematic skin type and the area where the index exceeds the set value are obtained, and the area is used as the recommended area.

[0044] Preferably, the content to be enhanced displayed exists in the form of one of image mark, text, video, and audio, or in a combination of at least two of the forms.

[0045] Preferably, the designated light source includes white light, polarized light or UV light.

[0046] Preferably, the problematic skin type includes at least one of skin color, wrinkles, spots, acne, blackheads, moisture content, and oil secretion.

[0047] In another aspect, an electronic device includes:

[0048] A full face image acquisition module, used to obtain a full face image;

[0049] A recommended area enhancement module is used to perform skin detection on the full face image, obtain the problematic skin type and the area where it is located, use the area where it is located as a recommended area, and enhance the display of the recommended area;

[0050] A problem skin type acquisition module, used to acquire the problem skin type corresponding to the recommended area when the local image acquisition device is captured to contact the recommended area;

[0051] A local image acquisition module, used to control the local image acquisition device to acquire a local image under a specified light source based on the acquired problematic skin type;

[0052] The local skin detection module is used to receive the local image sent by the local image acquisition device and detect problematic skin based on the local image.

[0053] Preferably, the electronic device further comprises: a collection area acquisition module, which is used to acquire the current collection area of ​​the local image collection device in real time and perform enhanced display on the current collection area.

[0054] On the other hand, an AR-based skin detection system includes: an electronic device and a local image acquisition device; the electronic device is used to implement the AR-based skin detection method; the local image acquisition device is connected to the electronic device, and is used to turn on the corresponding light source based on the problem skin type to capture the local image.

[0055] Compared with the prior art, the present invention has the following beneficial effects:

[0056] (1) The present invention detects problematic skin types based on a full-face image, enhances the display of a recommended area on the full-face image, and then guides a local image acquisition device to quickly align with the problematic skin, and turns on different light sources based on different problematic skin types to perform more detailed, accurate, and rapid detection of the problematic skin;

[0057] (2) The present invention can obtain the current acquisition area of ​​the local image acquisition device in real time and enhance the display of the current acquisition area, so that the user can more intuitively observe the position of the local image acquisition device on the face, thereby more accurately moving the local image acquisition device to the recommended area;

[0058] (3) The present invention locates the current coordinate position of the local image acquisition device based on at least one of the coordinate position of the front-end preset components of the local image acquisition device and the coordinate position of the key points of the hand, and the positioning is simple and fast.

[0059] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention so that it can be implemented in accordance with the contents of the specification, and in order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below.

[0060] Based on the following detailed description of specific embodiments of the present invention in conjunction with the accompanying drawings, those skilled in the art will become more aware of the above and other objects, advantages and features of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 This is a flow chart of the AR-based skin detection method according to the first embodiment of the present invention;

[0062] Figure 2 This is a flowchart of the use of the AR-based skin detection method according to the first embodiment of the present invention;

[0063] Figure 3 This is a flow chart of determining the current acquisition area according to the first embodiment of the present invention;

[0064] Figure 4 This is a schematic diagram of dividing regions using a facial key point detection method according to Embodiment 1 of the present invention;

[0065] Figure 5 This is a schematic diagram of a process of controlling a local image acquisition device to acquire a local image under a specified light source according to the first embodiment of the present invention;

[0066] Figure 6 A highlight image of oil secretion extracted based on a full face image according to the first embodiment of the present invention;

[0067] Figure 7 The enhanced image of the recommended oil secretion area extracted based on the full face image according to the first embodiment of the present invention;

[0068] Figure 8 The grease reflection image and highlight image are extracted by the local image acquisition device of the first embodiment of the present invention under white light; wherein (a) represents the grease reflection image, and (b) represents the grease highlight image;

[0069] Fig. 9 A feature map of blackheads extracted using a high-contrast algorithm based on a full-face image according to the first embodiment of the present invention;

[0070] Fig.10The highlight points of the blackheads are extracted after threshold segmentation based on the full face image in the first embodiment of the present invention;

[0071] Fig.11 This is an enhanced image of blackheads extracted based on a full-face image according to the first embodiment of the present invention;

[0072] Fig.12 This is a diagram showing the detection results of blackheads extracted by the local image acquisition device under UV light according to the first embodiment of the present invention;

[0073] Fig.13 The enhanced image of pigmented acne extracted based on the full face image according to the first embodiment of the present invention;

[0074] Fig.14 This is a detection result diagram extracted by the local image acquisition device of the first embodiment of the present invention under polarized light;

[0075] Fig.15 This is a structural block diagram of an electronic device according to a first embodiment of the present invention;

[0076] Fig.16 This is a structural block diagram of an AR-based skin detection system according to a second embodiment of the present invention;

[0077] Fig.17 It is a cross-sectional view of a local image acquisition device according to the second embodiment of the present invention. DETAILED DESCRIPTION

[0078] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention; it is obvious that the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments, and all other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making creative work are within the scope of protection of the present invention.

[0079] In the description of the present invention, it should be noted that the terms "comprises", "includes" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0080] In the description of the present invention, it should be noted that, unless otherwise clearly stipulated and limited, the terms "installed", "provided with", "mounted / connected", "connected", etc. should be understood in a broad sense. For example, "connected" can be a fixed connection, a detachable connection, or an integral connection. It can be a mechanical connection or an electrical connection. It can be a direct connection or an indirect connection through an intermediate medium. It can be the internal connection of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0081] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, step identifiers S101, S102, S103, etc. are only used for convenient expression and do not represent the execution order, and the corresponding execution order can be adjusted.

[0082] Embodiment 1

[0083] See also Figure 1 As shown, this embodiment provides an AR-based skin detection method, including:

[0084] S101, obtaining a full face image;

[0085] S102, performing skin detection on the full face image, obtaining problematic skin types and areas where the problematic skin types are located, taking the areas where the problematic skin types are located as recommended areas, and enhancing and displaying the recommended areas;

[0086] S103, when it is captured that the local image acquisition device contacts the recommended area, obtaining the problematic skin type corresponding to the recommended area;

[0087] S104, based on the acquired problematic skin type, controlling a local image acquisition device to acquire a local image under a designated light source;

[0088] S105, receiving the local image sent by the local image acquisition device, and detecting problematic skin based on the local image.

[0089] The execution subject of the above S101 to S105 is an electronic device with an image acquisition component, such as a mobile phone, a tablet computer, etc., which can be specifically configured as needed and is not limited in this embodiment.

[0090] In this embodiment, before capturing that a local image acquisition device has touched the recommended area, the method further includes:

[0091] The current acquisition area of ​​the local image acquisition device is acquired in real time, and the current acquisition area is enhanced and displayed.

[0092] Specifically, the content to be enhanced and displayed exists in the form of one of image mark, text, video, and audio, or in a combination of at least two of the forms.

[0093] The current acquisition area of ​​the local image acquisition device is acquired in real time, and the current acquisition area is enhanced and displayed, so that the user can more intuitively observe the position of the local image acquisition device on the face, and thus more accurately move the local image acquisition device to the recommended area.

[0094] For details, see Figure 2 As shown, the use process of the AR-based skin detection method of the present invention is as follows.

[0095] 1. Connect the local image acquisition device (such as a handheld skin test pen, etc.) and the electronic device (such as a mobile phone or tablet, etc.) via Wifi. First, the user turns on the Wifi function of the electronic device, searches for and selects the Wifi hotspot provided by the local image acquisition device.

[0096] 2. After the connection is successful, the user opens the skin detection app, which can connect to the local image acquisition device via Wifi to obtain the local image captured by the local image acquisition device. The user can select the required function on the app, such as selecting the problem skin type to be detected, the detection method, or turning on the corresponding detection mode, etc.

[0097] 3. After completing the connection between the electronic device and the local image acquisition device, first use the front camera of the electronic device to detect the entire face, obtain the scores of various dimensions of the face (skin color, wrinkles, spots, acne, blackheads, moisture content, and oil secretion), and give targeted recommendations for more serious areas (indicators greater than preset values) that need to be further detected using local image acquisition devices, and then perform real-time AR rendering to highlight the recommended areas.

[0098] The front camera of an electronic device is used to detect the entire face and obtain scores for various dimensions of the face (skin color, wrinkles, spots, acne, blackheads, moisture content, and oil secretion). This can be achieved using any method of the prior art, and this embodiment will not be described in detail.

[0099] 4. The electronic device performs AR rendering of the human face in real time, cooperates with the local image acquisition device to locate the user's current acquisition area, and renders it on the electronic device.

[0100] 5. After the user uses the current acquisition area to align with the recommended area, the local image acquisition device will be automatically controlled to acquire a local image under the specified light source and transmit it to the electronic device.

[0101] 6. After receiving the collected local image, the electronic device performs real-time processing and analysis. According to the preset algorithm, it detects and analyzes multiple types of skin blackheads, oil content, spots, acne, etc., and displays the results on the electronic device display.

[0102] 7. After the test results are displayed, users can analyze and compare the results to further understand their skin condition and choose appropriate skin care products and plans based on the test results.

[0103] 8. It can also record the results of each test and generate corresponding reports and analysis data to facilitate users to track and manage their skin health in the long term.

[0104] In this embodiment, when the local image acquisition device is captured to contact the recommended area, obtaining the problematic skin type corresponding to the recommended area specifically includes:

[0105] When the current acquisition area of ​​the captured local image acquisition device is located in the recommended area, the problematic skin type corresponding to the recommended area is obtained. If it is located in a non-recommended area, the electronic device or the local image acquisition device can issue a prompt, which can be a light prompt or a voice prompt, to prompt the user that it is a non-recommended area, or the local image acquisition device will stop the next step: that is, collect a local image under a specified light source and transmit it to the electronic device.

[0106] In one implementation, a method for acquiring a current acquisition area of ​​a local image acquisition device includes:

[0107] Acquiring a facial image including a local image acquisition device;

[0108] Identify the coordinate position of the front-end preset component of the local image acquisition device in the face image;

[0109] The area where the coordinate position is located is used as the current collection area.

[0110] It should be noted that the facial image described here and below can be a full face image or a partial face image, which is specifically set according to the user's usage habits and is not limited in this embodiment.

[0111] Specifically, when the user uses it, the front camera of the electronic device is turned on to collect the face image in real time. When the user holds the partial image acquisition device in contact with the face, the coordinate position of the front-end preset component of the partial image acquisition device in the face image is identified. It should be noted that the face image in this step needs to include the partial image acquisition device. Among them, the front-end preset component of the partial image acquisition device can be any component that is not blocked by the head, or a specially set positioning mark. It is set according to specific needs and is not limited in this embodiment.

[0112] Correspondingly, the area where the coordinate position is located can also be set as needed. Only the coordinate position can be taken, or the area where the coordinate position is located can be set according to the positional relationship between the front-end preset components and the detection head of the local image acquisition device. This embodiment does not impose any restrictions.

[0113] In another implementation, a method for acquiring a current acquisition area of ​​a local image acquisition device includes:

[0114] Acquiring a facial image including a handheld local image acquisition device;

[0115] Identify the coordinate position of a certain hand key point in the face image, or identify the coordinate position of the center of two or more hand key points in the face image;

[0116] The area where the coordinate position is located is used as the current collection area.

[0117] Specifically, when the user uses it, the front camera of the electronic device is turned on to collect facial images in real time. When the user holds the partial image acquisition device in contact with the face, the coordinate position of the key points of the hand holding the partial image acquisition device in the facial image is identified (using the hand key point detection algorithm of the prior art). It should be noted that the facial image in this step needs to include the hand holding the partial image acquisition device. Among them, the hand key point can be a certain hand key point such as the middle finger, or multiple hand key points such as the middle finger and thumb, etc., which are set according to specific needs and are not limited in this embodiment.

[0118] Correspondingly, the area where the coordinate position is located can also be set as needed. Only the coordinate position can be taken, or the area where the coordinate position is located can be set according to the positional relationship between the hand key point and the detection head of the local image acquisition device. This embodiment does not impose any restrictions.

[0119] In other implementations, a method for acquiring a current acquisition area of ​​a local image acquisition device includes:

[0120] Acquiring a facial image including a handheld local image acquisition device;

[0121] Identify the coordinate position of the front-end preset component of the local image acquisition device in the face image, and identify the coordinate position of a key point of a hand in the face image;

[0122] Calculate the coordinate position of the center of the coordinate position of the front-end preset component and the coordinate position of a certain hand key point;

[0123] The area where the center coordinates are located is taken as the current acquisition area.

[0124] Alternatively, a method for acquiring a current acquisition area of ​​a local image acquisition device includes:

[0125] Acquiring a facial image including a handheld local image acquisition device;

[0126] Identify the coordinate positions of the front-end preset components of the local image acquisition device in the face image, and identify the coordinate positions of two or more key points of the hand in the face image;

[0127] Calculate the coordinate position of the front-end preset component and the coordinate position of the center of the coordinate positions of two or more hand key points;

[0128] The area where the center coordinates are located is taken as the current acquisition area.

[0129] See also Figure 3 As shown, in order to obtain the acquisition area by combining the positioning identifier of the local image acquisition device with the hand key points, the positioning identifier is used in combination with the hand key points. The key points of the identified fingertips can be used to assist in determining the user's current acquisition area, which can make the positioning more accurate. In addition, by adding hand key point detection, it can effectively avoid the inability to judge the current acquisition area due to the obstruction of the front-end positioning identifier of the local image acquisition device.

[0130] For more details, see Figure 3 As shown, the method for locating the current coordinate position includes: using the front camera of the mobile phone to detect the face in real time through a preset face key point detection algorithm, and dividing the face into 16 areas according to the key points (see Figure 4 As shown), the contour coordinates Xi (i=1, 2, 3...16) of each of the 16 areas are obtained. When the local image acquisition device is close to the skin for detection, the positioning mark at the front end of the local image acquisition device is identified, so that the current coordinate Y of the positioning mark can be obtained, and the coordinates (F and T) of the middle finger and thumb are determined by identifying the key points of the hand, and then the centers of the three coordinates Y, F and T are calculated as the current detection position center G, and it is determined in which contour of Xi G falls, so that the current detection position can be determined.

[0131] Furthermore, after identifying the current acquisition area of ​​the local image acquisition device, the recommended area where the current acquisition area is located can be obtained, and the problem skin type corresponding to the recommended area can be obtained. The light source and detection algorithm are automatically selected for detection according to the problem skin type. For details, see Figure 5 Specifically, for the detection of oil, moisture, wrinkles and skin color, the white light source is automatically turned on; for the detection of spots and acne, the polarized light is automatically turned on; for the detection of blackheads, the UV light is automatically turned on.

[0132] The following will take oil detection, blackhead detection, spots and acne dimensions as examples to introduce in detail the methods and processes based on full-face image detection and local image detection.

[0133] (I) Grease detection

[0134] (1) Use electronic equipment to detect the collected full-face image

[0135] When using a full face image to estimate facial oil, because the facial oil part will appear as a bright area compared to other surrounding areas under white light, the following steps can be used to obtain a preliminary estimate of the oil area.

[0136] First, the original color image (raw) is converted to a grayscale image (gray) and the YCbCr color space (ycrcb).

[0137] Decompose the YCbCr color space into three channels and store them in a vectorcv::Mat type variable channels, where channel 0 represents brightness (Y), channel 1 represents blue chroma (Cb), and channel 2 represents red chroma (Cr).

[0138] Gaussian filtering is performed on the grayscale image to obtain a smoothed image (gs). Gaussian filtering is a commonly used image smoothing method that can effectively remove noise and details, making subsequent processing more accurate.

[0139] Select the Cr channel in the YCbCr color space and store it in the variable cr. The Cr channel represents the red chromaticity component in the image.

[0140] See also Figure 6 As shown in the figure, a new image (canvas) is obtained by subtracting the grayscale image (gs) from the Cr channel. This operation will suppress the red high brightness part in the image, making the oily highlight area of ​​the skin and the light and dark changes of the skin more obvious.

[0141] After binarization of the subtracted image (canvas), median filtering is used to remove discrete highlights to obtain a binary image (thres). Binarization is the process of converting the pixel values ​​in an image to 0 or 255, which is used to segment the target object and the background.

[0142] Finally, the binary image (thres) is subjected to Gaussian filtering to further smooth the image and remove discrete pixels, and then a binary image of the oily part of the skin can be obtained, see Figure 7 As shown in the figure, by counting the grease ratio (ratio) of each area and setting the threshold (ratio_grease) greater than 0.65, it is recommended that the area has severe grease.

[0143] (2) Use local images collected by local image acquisition equipment for detection

[0144] If it is detected that the local image acquisition device is performing detection in the corresponding area, the bio-impedance probe is started to perform oil detection, and the camera is started to take pictures using white light.

[0145] In the detection and analysis of the oil content of the skin, a local image acquisition device equipped with a bioimpedance probe is used for detection. The circuit inside the bioimpedance probe is composed of a power supply, a current limiting resistor and a measuring voltmeter. When the bioimpedance probe is attached to the surface of the skin, the current will pass through the skin and flow through the current limiting resistor, limiting the magnitude of the current, thereby preventing the current from causing excessive stimulation to the skin. At the same time, the voltmeter will measure the magnitude of the transcutaneous current and transmit the data to the acquisition device for processing and analysis.

[0146] After obtaining the bioimpedance data within a certain period of time, a series of processing steps are required to obtain a valid resistance value for the next step of data analysis. These steps include removing noise data, filtering, and calculating the final bioimpedance value.

[0147] The specific processing steps are as follows: First, remove the undetected resistance values ​​from the acquired bioimpedance data to exclude noise data. In this way, a valid resistance value list is obtained. Next, the valid resistance value list is processed by applying the resistance value filtering algorithm to obtain the final bioimpedance value. The step of the filtering algorithm is to obtain the mode interval in the list and calculate the average value in the interval. This average value is the final bioimpedance value A, and then it is segmented with the acquired local white light image through the threshold and the proportion is statistically calculated to obtain the oil ratio ratio (B). The specific oil value is fitted using a linear regression algorithm, y = w1*A+w2*B+b. The w values ​​are as follows: [[0.45,0.16]], and the b intercept value is: [0.43].

[0148] The grease reflection image and highlight image extracted by the local image acquisition device under white light can be found in Figure 8 As shown, (a) represents the grease reflection image, and (b) represents the grease highlight image.

[0149] (II) Blackhead detection

[0150] (1) Use electronic equipment to detect the collected full-face image

[0151] In the detection of blackheads, the Gaussian filter algorithm is first used to smooth the image and reduce the image noise, and then the high-contrast algorithm is used on the original image after Gaussian filtering. The high-contrast algorithm can be obtained by subtracting the Gaussian filtered image from the original image, because the Gaussian filtered image will retain the low-frequency features and filter out the high-frequency features. After subtraction, only the high-frequency blackhead details are retained. See Fig. 9As shown, after threshold processing, the mask image is obtained, in which the white points are blackheads, see Fig.10 shown.

[0152] See also Fig.11 As shown in the figure, after the above series of steps, the preliminary distribution of blackheads can be obtained, the number of blackheads in each area can be counted, and the number threshold k is set. When k>120, it is recommended to the user that the blackhead dimension in this area is serious. Then, the local image acquisition device is used to perform more detailed blackhead detection on the local skin.

[0153] (2) Use local images collected by local image acquisition equipment for detection

[0154] If it is detected that the local image acquisition device is used to detect this area, the UV light and camera of the local image acquisition device are started to take pictures.

[0155] The local skin image collected by the local image acquisition device is collected using UV light. Since the reflectivity of blackheads under ultraviolet light is different, the characteristics of blackheads can be highlighted by shooting with UV light, which is convenient for detection. The nanodet target detection network is used to detect blackheads. The following is an overview of this process.

[0156] 1. Preprocess the collected UV light images, including denoising and sharpening, and use the nearest neighbor interpolation method to adjust the image size and scale to adapt to the image size required by the nanodet network.

[0157] 2. Design nanodet network. NanoDet network structure is a network structure based on FCOS (Fully Convolutional One-Stage). The structure mainly includes four stages: Backbone network, Feature Fushion network, Subnet network and Head network. Use EfficientNet as its Backbone network. EfficientNet is an efficient CNN structure that improves network performance by using a composite scaling method to adjust depth, width and resolution at the same time, and can better extract feature information in the image. The Feature Fusion network is responsible for fusing the multi-scale feature maps extracted by the Backbone network to generate richer and more discriminative feature representations. BiFPN (Bidirectional Feature Pyramid Network) is used as the Feature Fusion network. It is a feature pyramid network that can process feature maps of different scales at the same time, effectively improving the detection performance of the model. The Subnet network is a subnet used to generate a detection box. Two independent Subnet networks are used, one for predicting the horizontal detection box and the other for predicting the rotational detection box, so that the rotating target can be detected more accurately. The Head network is used to classify and regress the detection box. It consists of two branches, one for predicting the class probability of the target and the other for predicting the coordinate offset of the target box. FocalLoss is used as the loss function of the Head network, which can effectively solve the problem of class imbalance and improve the training effect of the model.

[0158] 3. Label the collected and preprocessed images and send them to the constructed network model for training to obtain the model.

[0159] 4. In post-processing, the detection results are obtained through forward inference calculation of the model, see Fig.12 As shown, the following algorithm is used to obtain the density of bounding boxes and fit the relationship between the number and density and the score. The average distance between bounding boxes is calculated as the density index. The average distance can be calculated using the following formula:

[0160]

[0161] Among them, n represents the number of bounding boxes, (x i ,y i ) represents the center coordinates of the i-th boundingbox.

[0162] The score can be calculated using the following formula:

[0163] Score=100-k×nm×AvgDistance

[0164] Among them, k and m are constants and can be adjusted according to specific circumstances.

[0165] (III) Pigmentation and acne detection

[0166] (1) Use electronic equipment to detect the collected full-face image

[0167] The detection of spots and acne adopts a deep algorithm solution. RetinaNet is a target detection algorithm based on Focal Loss. It solves the problem of class imbalance by changing the weight of the loss function, so that the model can better detect small targets and difficult samples. When using RetinaNet to detect spots and acne on the face of a person, there are mainly the following steps:

[0168] 1. First, you need to prepare training sets and test sets, which contain facial image data with spots and acne. For each sample, you need to manually annotate the spots and acne areas it contains so that the algorithm can learn and detect.

[0169] 2. Use the prepared dataset to train the RetinaNet model. During the training process, RetinaNet will extract image features through multiple convolutional and fully connected layers, and use mechanisms such as Anchor Box and Focal Loss to perform target detection. After the training is completed, a model for facial spot and acne detection can be obtained.

[0170] 3. Use the prepared test set to test and evaluate the trained model. For each test image, RetinaNet will output the coordinates and confidence of the spots and acne areas it contains. The final detection results can be filtered out by adjusting the threshold. Some indicators such as precision and recall can be used to evaluate the performance of the model. In terms of mathematical principles, RetinaNet uses a loss function called Focal Loss to solve the problem of class imbalance in target detection. Specifically, FocalLoss will reduce the weight of samples that are easy to classify, thereby paying more attention to samples that are difficult to classify, so that the model pays more attention to samples in minority categories during training. The formula is as follows:

[0171] FL(p t )=-α t (1-p t ) γ log(p t )

[0172] Among them, p t represents the probability predicted by the model, α t Represents the weight adjustment factor, and γ represents the adjustment factor. Focal Loss can be used in RetinaNet to adjust the imbalance of positive and negative sample ratios in target detection, thereby improving the performance of the model.

[0173] In addition, RetinaNet also uses mechanisms such as Anchor Box and Feature Pyramid to achieve target detection. Anchor Box is a preset bounding box that can cover different sizes and shapes of target objects, thereby improving the generalization ability of the model. Feature Pyramid is a multi-scale feature extraction method that can extract image features at different scales and combine AnchorBox for target detection, so that target objects of different sizes can be detected.

[0174] Then, the number of spots or acne detected by RetinaNet is counted, see Fig.13 That is, the result graph of the model forward reasoning detection is used to determine whether the spots and acne in the area are serious through the preset number threshold k. When k>=2 for a single area, this dimension can be recommended.

[0175] (2) Use local images collected by local image acquisition equipment for detection

[0176] If it is detected that the local image acquisition device corresponds to this area, the polarized light and camera of the local image acquisition device are started to take pictures.

[0177] In the detection of spots and acne, local image acquisition equipment is used and polarized light mode is adopted for acquisition. Polarized light can reduce light interference and scattering and improve imaging clarity. For the segmentation of spots and acne, the U-Net network is used for segmentation, and some modifications are made to make it more suitable for the spot segmentation task. The residual block is introduced into the U-Net model to enhance the feature extraction ability of the model. At the same time, the size of the convolution kernel is set to 3×3 to further improve the receptive field of the model. In addition, the attention mechanism is added to make the model pay more attention to the spots and acne areas. The SENet method is used to improve the performance of the model by adding an attention module.

[0178] For the areas segmented from spots and acne, see Fig.14 As shown in the figure, an algorithm based on color statistics can be used to assess its severity. Taking spots as an example, the calculation method for acne is the same, and the specific algorithm is as follows:

[0179] 1. Convert the spot segmentation area into HSV color space.

[0180] 2. Count the distribution of pixel values ​​of the three channels H, S, and V in the color spot area, and calculate the mean and standard deviation.

[0181] For the H channel, if the color range is between red and yellow, a certain score is given. The specific score can be calculated by the mean and standard deviation of the H channel. The calculation formula is:

[0182]

[0183] Among them, R mean and Y mean Represents the mean of H channel for red and yellow, G mean and G std Represent the mean and standard deviation of the H channel in the spot area respectively.

[0184] For the S and V channels, if the pixel value is above a certain threshold, a certain score is given. The specific score can be calculated by the mean and standard deviation of the S and V channels. The calculation formula is:

[0185]

[0186]

[0187] Among them, S thres and V thres is the threshold of S and V channels, S mean and V mean are the mean values ​​of S and V channels respectively, S std and V std are the standard deviations of the S and V channels, respectively.

[0188] Finally, the three channel scores are weighted and summed to get the final score:

[0189] Score total =ω H ×ScoreH+ω S ×ScoreS+ω V ×ScoreV

[0190] Among them, ω H ,ω S ,ω V is the weight of the H, S, and V channel scores, which can be adjusted experimentally.

[0191] The present invention detects problem skin types based on a full-face image, enhances the display of a recommended area on the full-face image, and then guides a local image acquisition device to quickly align with the problem skin, and turns on different light sources based on different problem skin types to perform more detailed, accurate, and rapid detection of the problem skin.

[0192] See also Fig.15 As shown, this embodiment also discloses an electronic device, including:

[0193] Full face image acquisition module 1501, used to acquire full face images;

[0194] The recommended area enhancement module 1502 is used to perform skin detection on the full face image, obtain the problematic skin type and the area where it is located, use the area where it is located as the recommended area, and enhance the recommended area for display;

[0195] The problem skin type acquisition module 1503 is used to acquire the problem skin type corresponding to the recommended area when the local image acquisition device is captured to contact the recommended area;

[0196] The local image acquisition module 1504 is used to control the local image acquisition device to acquire a local image under a specified light source based on the acquired problematic skin type;

[0197] The local skin detection module 1505 is used to receive the local image sent by the local image acquisition device, and detect problematic skin based on the local image.

[0198] Specifically, the electronic device further includes: a collection area acquisition module, which is used to acquire the current collection area of ​​the local image collection device in real time and perform enhanced display on the current collection area.

[0199] The specific implementation of each module of an electronic device refers to a skin detection method based on AR, and will not be repeated in this embodiment.

[0200] Embodiment 2

[0201] See also Fig.16 As shown, this embodiment discloses an AR-based skin detection system, including: an electronic device 161 and a local image acquisition device 162; the electronic device 161 is used to implement the AR-based skin detection method described in Example 1; the local image acquisition device 162 is connected to the electronic device 161, and is used to turn on the corresponding light source based on the problem skin type to capture the local image.

[0202] See also Fig.17As shown, it is a cross-sectional view of the local image acquisition device of this embodiment, including a detection head 1621, a positioning mark 1622, a light source board 1623, a polarizer 1624, an image acquisition component 1625 and a control mainboard 1626. The current acquisition area can be obtained based on the positioning mark 1622. The light source board 1623 is used to emit white light and UV light. The control mainboard 1626 is used to receive instructions from the electronic device 161, control the light source board 1623 or the polarizer 1624 to turn on, and control the image acquisition component 1625 to capture a local image.

[0203] The above is only a preferred specific implementation of the present invention; however, the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical solution and its improved conception within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A skin detection method based on AR, characterized in that: include: Get a full face image; Perform skin detection on the full-face image to obtain the problematic skin type and the area where it is located, use the area where it is located as a recommended area, and enhance the display of the recommended area; When it is captured that the local image acquisition device contacts the recommended area, the problematic skin type corresponding to the recommended area is obtained; Based on the acquired problematic skin type, controlling the local image acquisition device to acquire the local image under the specified light source; The local image sent by the local image acquisition device is received, and the problematic skin is detected based on the local image.

2. The AR-based skin detection method according to claim 1, characterized in that: Before capturing that a local image acquisition device has touched the recommended area, the method further includes: The current acquisition area of ​​the local image acquisition device is acquired in real time, and the current acquisition area is enhanced and displayed.

3. The AR-based skin detection method according to claim 1, characterized in that: When the local image acquisition device is captured to contact the recommended area, obtaining the problematic skin type corresponding to the recommended area specifically includes: When it is captured that the current acquisition area of ​​the local image acquisition device is located in the recommended area, the problematic skin type corresponding to the recommended area is obtained.

4. The AR-based skin detection method according to claim 2, characterized in that: A method for acquiring a current acquisition area of ​​a local image acquisition device includes: Acquiring a facial image including a local image acquisition device; Identify the coordinate position of the front-end preset component of the local image acquisition device in the face image; The area where the coordinate position is located is used as the current collection area.

5. The AR-based skin detection method according to claim 2, characterized in that: A method for acquiring a current acquisition area of ​​a local image acquisition device includes: Acquiring a face image including a handheld local image acquisition device; Identify the coordinate position of a certain hand key point in the face image, or identify the coordinate position of the center of two or more hand key points in the face image; The area where the coordinate position is located is used as the current collection area.

6. The AR-based skin detection method according to claim 2, characterized in that: A method for acquiring a current acquisition area of ​​a local image acquisition device includes: Acquiring a face image including a handheld local image acquisition device; Identify the coordinate position of the front-end preset component of the local image acquisition device in the face image, and identify the coordinate position of a key point of a hand in the face image; Calculate the coordinate position of the center of the coordinate position of the front-end preset component and the coordinate position of a certain hand key point; The area where the center coordinates are located is taken as the current acquisition area.

7. The AR-based skin detection method according to claim 2, characterized in that: A method for acquiring a current acquisition area of ​​a local image acquisition device includes: Acquiring a face image including a handheld local image acquisition device; Identify the coordinate positions of the front-end preset components of the local image acquisition device in the face image, and identify the coordinate positions of two or more key points of the hand in the face image; Calculate the coordinate position of the front-end preset component and the coordinate position of the center of the coordinate positions of two or more hand key points; The area where the center coordinates are located is taken as the current acquisition area.

8. The AR-based skin detection method according to any one of claims 4 to 7, characterized in that: The method for defining the area where the coordinate position is located includes: Based on the acquired face image, the face key point detection method is used to divide the face into preset areas according to the face key points; Get the contour coordinates of each region; Determine the contour to which the coordinate position belongs; Use the corresponding contour as the current acquisition area.

9. The AR-based skin detection method according to claim 1, characterized in that: Obtain the problematic skin type and area, and use the area as the recommended area, including: The problematic skin type and the area where the index exceeds the set value are obtained, and the area is used as the recommended area.

10. The AR-based skin detection method according to claim 1, characterized in that: The content to be enhanced and displayed exists in the form of one of image mark, text, video, and audio, or in a combination of at least two of the forms.

11. The AR-based skin detection method according to claim 1, characterized in that: The designated light source includes white light, polarized light or UV light.

12. The AR-based skin detection method according to claim 1, characterized in that: The problematic skin type includes at least one of skin color, wrinkles, spots, acne, blackheads, moisture content, and oil secretion.

13. An electronic device, characterized in that: include: A full face image acquisition module, used to obtain a full face image; A recommended area enhancement module is used to perform skin detection on the full face image, obtain the problematic skin type and the area where it is located, use the area where it is located as a recommended area, and enhance the display of the recommended area; A problem skin type acquisition module, used to acquire the problem skin type corresponding to the recommended area when the local image acquisition device is captured to contact the recommended area; A local image acquisition module, used to control the local image acquisition device to acquire a local image under a specified light source based on the acquired problematic skin type; The local skin detection module is used to receive the local image sent by the local image acquisition device and detect problematic skin based on the local image.

14. The electronic device according to claim 13, characterized in that: Also includes: The acquisition area acquisition module is used to acquire the current acquisition area of ​​the local image acquisition device in real time and perform enhanced display on the current acquisition area.

15. A skin detection system based on AR, characterized in that: include: An electronic device and a local image acquisition device; the electronic device is used to implement the method described in any one of claims 1 to 12; the local image acquisition device is connected to the electronic device and is used to capture a local image by turning on a corresponding light source based on the problem skin type.

Images