A skin care effect detection method and detection device based on deep learning technology

Abstract

The present invention discloses a skin care effect detection method and detection device based on deep learning technology, wherein the detection method comprises: obtaining an initial skin image before care; segmenting the initial skin image to generate detection units; training a detection model; using the detection model to detect each detection unit to determine the position and initial features of facial feature points; obtaining a nursing skin image after care; matching the nursing skin image with the initial skin image, determining the corresponding position of the facial feature points on the nursing skin image according to the position information of the facial feature points and again using the detection model to detect the facial feature points on the nursing skin image to obtain the state after care; comparing the initial features and post-care features of the facial feature points and generating a care report. The present invention can make accurate measurements of the skin care effect and give an objective evaluation by comparing and analyzing the differences in skin images before and after skin care based on deep learning technology.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a skin care effect detection method and detection device based on deep learning technology. Background Art

[0002] As people's living standards improve, after meeting basic needs for food, clothing, housing and transportation, people's demand for improving their external image has become increasingly prominent, especially for facial skin care, forming a huge consumer market.

[0003] Skin care forms are ranked from low to high in terms of professionalism: daily care at home, professional care in beauty salons, and medical beauty. Different products and different skin care methods have different effects, but there has been no accurate method for evaluating skin care effects on the market.

[0004] The skin testing instruments currently on the market are divided into two types based on the testing method: local testing and full-face testing.

[0005] Local testing uses a method that magnifies local skin at a high level, which has a higher degree of accuracy. However, the facial skin in different locations may be quite different, and it is difficult to locate the same testing location twice, making it impossible to accurately compare the effects before and after skin care.

[0006] The full-face detection is composed of various types of magic mirrors. The detection data is relatively comprehensive, but the accuracy is generally insufficient. For example, for the same person, if two tests are performed consecutively at the same time, the data deviation before and after is often more than 5%. It is also impossible to accurately compare the effects before and after skin care.

[0007] With the development of mobile phone camera technology, the shooting accuracy of the rear camera of the mobile phone is getting higher and higher (up to 100 million pixels), and the clarity of the image can basically meet the needs of image analysis. At the same time, the deep learning technology in the field of artificial intelligence has ushered in great development, and various face recognition technologies are applied in various fields. The basic algorithm also provides a solid foundation for the detection accuracy of the present invention.

[0008] In view of this, it is necessary to propose a method that can easily and accurately measure and evaluate the differences before and after skin care. Summary of the invention

[0009] The purpose of the present invention is to provide a skin care effect detection method based on deep learning technology to solve the shortcomings of the prior art. It can accurately measure the skin care effect and give an objective evaluation by comparing and analyzing the differences in skin images before and after skin care based on deep learning technology.

[0010] A first aspect of the present invention provides a skin care effect detection method based on deep learning technology, comprising:

[0011] S1: Obtain the initial skin image before care;

[0012] S2: After preprocessing the initial skin image, perform matrix segmentation to generate N*N detection units;

[0013] S3: A deep learning-based detection model built through local facial feature recognition and convolutional network learning, and the detection model is trained with a sample dataset;

[0014] S4: Use the detection model to detect each detection unit to determine the location of facial feature points and initial features;

[0015] S5: obtaining a skin image after care;

[0016] S6: Matching the treated skin image with the initial skin image, determining the corresponding position of the facial feature point on the treated skin image according to the position information of the facial feature point, and detecting the facial feature point on the treated skin image again using the detection model to obtain the post-treatment state;

[0017] S7: Compare the initial features and post-care features of the facial feature points and generate a care report.

[0018] Optionally, step “S3: constructing a deep learning-based detection model through local facial feature recognition and convolutional network learning and training the detection model with a sample data set” specifically includes:

[0019] S31: Collect frontal images with skin problems respectively, mark facial feature points with skin problems in the frontal images, and form a sample dataset D1 of facial feature points of problem skin;

[0020] S32: Train the detection model, use the two-stage channel selection pre-training method, first train the backbone network ηmain from the ImageNet image classification dataset DCls; then train the instance segmentation model mpretrained from the lesion boundary segmentation dataset DLBS published by ISIC; on the problem skin facial feature point sample dataset D1, select the best two channels of the instance segmentation model mpretrained to obtain mpretrained_2; then train to obtain the detection model ηdetect.

[0021] Optionally, step “S4: using the detection model to detect each detection unit to determine the position of facial feature points and initial features” specifically includes:

[0022] S41: predict m target detection frames for each detection unit, predict the horizontal coordinate x, vertical coordinate y, width w, height h of the target region of each target detection frame, and reliability information c of the target detection frame;

[0023] S42: The detection model ηdetect detects each target detection frame and obtains the location of the problematic skin feature points and initial feature data.

[0024] Optionally, the reliability information c is calculated as follows:

[0025] C=P(o)*IOUt_p

[0026] Where P(o) indicates whether the target detection frame contains the target area. If there is a target area, it is 1; if there is no target area, it is 0. IOUt_p represents the overlap between the predicted target area and the actual target area. If there is no target area in the target detection frame, c is 0, otherwise c represents the intersection ratio between the predicted target area and the actual target area.

[0027] Optionally, step “S42: using the detection model ηdetect to detect each target detection frame and obtain the location of the problematic skin feature point and initial feature data” includes:

[0028] S421: The detection model ηdetect detects the segmented distribution map of problematic facial feature points and obtains the number Ns of problematic facial feature points, as well as the area Ai and position (xi, yi) of the i-th problematic facial feature point;

[0029] S422: Obtain the area Aj and position (xj, yj) of the facial feature point j adjacent to the i-th problematic facial feature point;

[0030] S423: Determine the initial features based on the area and position relationship between the i-th facial feature point and the adjacent j-th facial feature point, and determine the problem type of the problem skin feature point based on the initial features.

[0031] Optionally, step “S423: determining the initial features according to the area and position relationship between the i-th facial feature point and the adjacent j-th facial feature point and determining the problem type of the problematic skin feature point according to the initial features” includes:

[0032] S4231: If the ith facial feature point is linearly continuous with the adjacent jth facial feature point, and the length of the linear continuity of the adjacent facial feature points exceeds a first predetermined value K, it is indicated as a wrinkle and the number and length of the wrinkles are recorded;

[0033] S4232: If the ith facial feature point and the adjacent jth facial feature point are similar and discontinuous in a flake shape, and the RGB colors of the ith facial feature point and the jth facial feature point are in the range of yellow to dark brown, it indicates that there are spots in the area, and the size and color depth of the spots are recorded;

[0034] S4233: If the ith facial feature point and the adjacent jth facial feature point are approximately circular, and the diameter of the circle is less than 0.5 mm, then the area is a pore;

[0035] S4234: If the ith facial feature point and the adjacent jth facial feature point are continuous and approximately circular, and the diameter of the circle exceeds 1 mm, it is determined that the area is acne.

[0036] Optionally, step “S4: using the detection model to detect each detection unit to determine the position of facial feature points and initial features” further includes:

[0037] S43: Collect RGB color information of the detection unit, and evaluate the skin color of the detection unit area according to the color information.

[0038] Optionally, step “S5: obtaining a skin-treated image after treatment” specifically includes:

[0039] S51: Collect multiple face photos through a picture collector, including but not limited to frontal, left side, and right side photos;

[0040] S52: The multiple face photos collected above are combined into a complete face image through photo synthesis technology.

[0041] Optionally, “S6: matching the treated skin image with the initial skin image, determining the corresponding position of the facial feature point on the treated skin image according to the facial feature point position information, and again using the detection model to detect the facial feature point on the treated skin image to obtain the post-treatment state” includes the following steps:

[0042] S61: Acquire a skin care image;

[0043] S62: placing the treated skin image and the initial skin image in the same coordinate system;

[0044] S63: Matching the characteristic points on the treated skin image with the characteristic points on the initial skin image so as to match the treated skin image with the initial skin image.

[0045] A second aspect of the present invention provides a skin care effect detection device based on deep learning technology, comprising:

[0046] An image acquisition unit, used to acquire an initial skin image before care and an image of the skin after care;

[0047] An image preprocessing unit, used for performing matrix segmentation on the initial skin image after preprocessing to generate N*N detection units;

[0048] A detection unit, which is used to construct a deep learning-based detection model through local facial feature recognition and convolutional network learning and train the detection model with a sample data set, and use the detection model to detect each detection unit to determine the location of facial feature points and initial features;

[0049] An evaluation unit is used to match the nursing skin image with the initial skin image, determine the corresponding position of the facial feature point on the nursing skin image according to the position information of the facial feature point, and use the detection model to detect the facial feature point on the nursing skin image again to obtain the post-nursing state; compare the initial features and post-nursing features of the facial feature point and generate a nursing report.

[0050] A third aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the program implements the steps of any of the above-mentioned methods when executed by a processor.

[0051] A fourth aspect of the present invention provides a terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any of the above-mentioned methods when executing the program.

[0052] Compared with the existing technology, the present invention uses a mobile phone or other device to capture image information before and after care, and then uses a detection model obtained by a deep learning algorithm to perform detection to analyze and compare the skin characteristics before and after care to finally obtain a test report. This can obtain the skin effect after care in a timely, accurate and efficient manner, and is convenient to use. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 A schematic diagram of the process of a skin care effect detection method based on deep learning technology disclosed in an embodiment of the present invention is shown;

[0054] Figure 2 A schematic diagram of the detection model training process in the skin care effect detection method based on deep learning technology disclosed in an embodiment of the present invention is shown;

[0055] Figure 3 A schematic diagram showing a flow chart of a detection process in a skin care effect detection method based on deep learning technology disclosed in an embodiment of the present invention is shown;

[0056] Figure 4 A framework diagram of a skin care effect detection device based on deep learning technology disclosed in an embodiment of the present invention is shown;

[0057] Figure 5 A structural block diagram of a terminal 500 according to an embodiment of the present application is shown;

[0058] Figure 6A structural block diagram of a terminal 600 according to another embodiment of the present application is shown. DETAILED DESCRIPTION

[0059] The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, but should not be construed as limiting the present invention.

[0060] The first aspect of the present invention provides a skin care effect detection method based on deep learning technology.

[0061] As an embodiment of the first aspect, Figure 1 As shown, a skin care effect detection method based on deep learning technology includes:

[0062] S1: Acquire the initial skin image before treatment.

[0063] In this embodiment, before skin care, you can use a mobile terminal to take a photo of the customer's face, and you can use the camera module built into the system applet (or APP) to collect images with the help of the mobile phone camera. Follow the prompts in the system to take photos, take three photos of the front face, left face, and right face, and upload the photos to the cloud.

[0064] S2: After preprocessing the initial skin image, perform matrix segmentation to generate N*N detection units;

[0065] S3: A deep learning-based detection model is constructed through local facial feature recognition and convolutional network learning, and the detection model is trained with a sample dataset.

[0066] The sample data sets are respectively collected from sample data sets formed by different types of problematic skin, including wrinkle sample data sets, acne sample data sets, pore sample data sets, pore sample data sets and spot sample data sets. Each sample data set stores corresponding problematic skin images for model training.

[0067] Specifically, this step may include:

[0068] S31: Collect frontal images with skin problems respectively, mark facial feature points with skin problems in the frontal images, and form a sample dataset D1 of facial feature points of problem skin;

[0069] As described above, the facial feature points of skin problems can be classified into mainly wrinkles, acne, pores and spots and corresponding sample data sets are formed respectively, and the constructed detection model is trained with the corresponding sample data sets.

[0070] S32: Train the detection model, use the two-stage channel selection pre-training method, first train the backbone network ηmain from the ImageNet image classification dataset DCls; then train the instance segmentation model mpretrained from the lesion boundary segmentation dataset DLBS published by ISIC; select the best two channels of the instance segmentation model mpretrained on the problem skin facial feature point sample dataset D1 to obtain mpretrained_2; then train to obtain the detection model ηdetect. Figure 2 shown.

[0071] S4: Use the detection model to detect each detection unit to determine the location and initial features of facial feature points. Use the trained detection model to detect the image before care to determine the location and initial features of the problematic facial feature points in each detection unit and record the initial features.

[0072] Specifically, this step includes:

[0073] S41: predict m target detection frames for each detection unit, predict the horizontal coordinate x, vertical coordinate y, width w, height h of the target region of each target detection frame, and reliability information c of the target detection frame;

[0074] The reliability information c is calculated as follows:

[0075] C=P(o)*IOUt_p

[0076] Where P(o) indicates whether the target detection frame contains the target area. If there is a target area, it is 1; if there is no target area, it is 0. IOUt_p represents the overlap between the predicted target area and the actual target area. If there is no target area in the target detection frame, c is 0, otherwise c represents the intersection ratio between the predicted target area and the actual target area.

[0077] S42: The detection model ηdetect detects each target detection frame and obtains the location of the problem skin feature point and the initial feature data.

[0078] like Figure 3 As shown, step S42 may specifically include:

[0079] S421: The detection model ηdetect is used to detect and segment the distribution map of the problem facial feature points and obtain the number of the problem facial feature points Ns, as well as the area Ai and position (xi, yi) of the i-th problem facial feature point;

[0080] S422: Obtain the area Aj and position (xj, yj) of the facial feature point j adjacent to the i-th problematic facial feature point;

[0081] S423: Determine the initial features based on the area and position relationship between the i-th facial feature point and the adjacent j-th facial feature point, and determine the problem type of the problem skin feature point based on the initial features.

[0082] The above step "S423: determining the initial features according to the area and position relationship between the i-th facial feature point and the adjacent j-th facial feature point and determining the problem type of the problematic skin feature point according to the initial features" may include the following specific judgment process:

[0083] S4231: If the ith facial feature point is linearly continuous with the adjacent jth facial feature point, and the length of the linear continuity of the adjacent facial feature points exceeds the first predetermined value K, it is indicated as a wrinkle and the number and length of the wrinkles are recorded; wherein the first predetermined value K is 3mm. Whether the two are adjacent can be determined by the coordinates of the position of the problem facial feature point. When the difference between the coordinates of the two is small, it indicates that the two are continuous. When the difference between the coordinates of the two is large, it indicates that there is a gap between the two. When the coordinates of several consecutive adjacent facial feature points are continuously increasing, it can be indicated as a wrinkle.

[0084] S4232: If the ith facial feature point is similar to the adjacent jth facial feature point in a sheet-like manner and is discontinuous, and the RGB colors of the ith facial feature point and the jth facial feature point are in the yellow to dark brown range, it indicates that there are spots in the area, and the size and color depth of the spots are recorded. If the detection point is similar to the surrounding detection points in a sheet-like manner (irregular shape), and the RGB color is in the yellow to dark brown range, it can be regarded as a spot, and then it is judged as age spots, chloasma, freckles, etc. according to the size and color depth of the spots, and the system records the number and area of ​​the spots.

[0085] S4233: If the ith facial feature point and the adjacent jth facial feature point are approximately circular, and the diameter of the circle is less than 0.5 mm, then the area is a pore;

[0086] S4234: If the ith facial feature point and the adjacent jth facial feature point are continuous and approximately circular, and the diameter of the circle exceeds 1 mm, it is determined that the area is acne.

[0087] The above method can well determine the specific morphological features of each facial problem feature point and record the morphological features to form a pre-care facial inspection report.

[0088] Optionally, in step “S4: using the detection model to detect each detection unit to determine the position of facial feature points and initial features” further includes:

[0089] S4: Collect RGB color information of the detection unit, and evaluate the skin color of the detection unit area according to the color information.

[0090] The skin color of the corresponding detection unit can be determined according to the RGB color of the acquired image, and the final skin color can be determined according to the characteristics of the entire image.

[0091] After the treatment is completed, step "S5: obtain the treated skin image after treatment". The method of obtaining the image here is similar to the method of obtaining the image before treatment. Specifically, the customer's face is photographed by a mobile terminal. The image can be collected by the camera module built into the system applet (or APP) with the help of the mobile phone camera. Take photos according to the prompts in the system, take three photos of the front face, left face, and right face respectively, and upload the photos to the cloud.

[0092] Furthermore, "obtaining an initial skin image before care" specifically includes collecting multiple face photos through a picture collector, including but not limited to front, left and right side photos, and synthesizing the multiple face photos collected above into a complete face image through a photo synthesis technology. By synthesizing the acquired front, left and right side images, the accuracy of the image data can be better achieved.

[0093] S6: Matching the treated skin image with the initial skin image, determining the corresponding position of the facial feature point on the treated skin image according to the facial feature point position information, and detecting the facial feature point on the treated skin image again using the detection model to obtain the post-treatment state.

[0094] This step may specifically include:

[0095] S61: Acquire a skin care image;

[0096] S62: placing the treated skin image and the initial skin image in the same coordinate system;

[0097] S63: Matching the characteristic points on the treated skin image with the characteristic points on the initial skin image so as to match the treated skin image with the initial skin image.

[0098] S7: Compare the initial features and post-care features of the facial feature points and generate a care report.

[0099] The skin care effect detection method based on deep learning technology disclosed in the embodiment of the present invention uses a mobile phone or other device to capture image information before and after care, and then uses a detection model obtained by a deep learning algorithm to perform detection, analyze and compare skin characteristics before and after care, and finally obtain a detection report. It can obtain the skin effect after care in a timely, accurate and efficient manner, and is convenient to use.

[0100] In this embodiment, the feature points can be some facial feature parts such as eyes, nose, mouth or ears, or facial contour lines. Through the above feature points, the image before facial care and the image after facial care are overlapped so that the same positions are placed together, and then the corresponding positions are detected again according to the recorded positions of the facial feature points and the detection results are recorded, and a post-care detection report is generated according to the detection results, and a care evaluation report is finally generated by combining the post-care detection report and the pre-care detection report.

[0101] In the later stage, before and after the customer's care, continuous photo analysis can be carried out, and a skin change report can be generated according to the date of the photo, so as to track the customer's skin condition for a long time and provide targeted skin care. In this detection method, since different people are suitable for different skin types, each person has his or her own record after use. In this way, people with similar skin types are automatically clustered in the system, and a comparison report is automatically generated, which can conveniently analyze the effect of people with different skin types using this whitening product.

[0102] The second aspect of the present invention discloses a detection device. The device can implement the method of the above embodiment. The following device embodiment can be used to understand the above method embodiment. The above method embodiment can also be used to understand the following device embodiment.

[0103] Figure 4 Such a skin care effect detection device based on deep learning technology is shown, comprising:

[0104] An image acquisition unit, used to acquire an initial skin image before care and an image of the skin after care;

[0105] An image preprocessing unit, used for performing matrix segmentation on the initial skin image after preprocessing to generate N*N detection units;

[0106] A detection unit, which is used to construct a deep learning-based detection model through local facial feature recognition and convolutional network learning and train the detection model with a sample data set, and use the detection model to detect each detection unit to determine the location of facial feature points and initial features;

[0107] An evaluation unit is used to match the nursing skin image with the initial skin image, determine the corresponding position of the facial feature point on the nursing skin image according to the position information of the facial feature point, and use the detection model to detect the facial feature point on the nursing skin image again to obtain the post-nursing state; compare the initial features and post-nursing features of the facial feature point and generate a nursing report.

[0108] Those skilled in the art can clearly understand that the division of the "unit" in the embodiment of the present application is only a division of a logical function. There may be other division methods in actual implementation, for example, multiple "units" can be combined or integrated into one "unit" to realize the corresponding function. Or a "unit" can be decomposed into multiple units to realize the corresponding function together. The "unit" in the embodiment of the present application can be software and / or hardware that can independently complete or cooperate with other components to complete a specific function, wherein the hardware can be, for example, FPGA (Field-Programmable Gate Array), IC (Integrated Circuit), etc., which will not be repeated here one by one.

[0109] In a third aspect of the present invention, an embodiment of the present application provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the steps of the method of any of the above embodiments are implemented. The computer-readable storage medium may include, but is not limited to, any type of disk, including a floppy disk, an optical disk, a DVD, a CD-ROM, a micro drive, and a magneto-optical disk, a ROM, a RAM, an EPROM, an EEPROM, a DRAM, a VRAM, a flash memory device, a magnetic card or an optical card, a nano device (including a molecular memory IC), or any type of medium or device suitable for storing instructions and / or data.

[0110] In a fourth aspect of the present invention, an embodiment of the present application provides a terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any of the above-mentioned method embodiments when executing the program.

[0111] The terminal in the embodiments of the present application may include but is not limited to a smart tablet computer, a handheld computer, a personal digital assistant (PAD), a smart interactive tablet and other devices.

[0112] Figure 5 A structural block diagram of a terminal 500 according to an embodiment of the present application is shown.

[0113] See also Figure 5 The terminal includes a processor 510, a memory 520, a camera 530 and a microphone 540.

[0114] In the embodiment of the present application, the processor 510 is the control center of the computer system, which can be a processor of a physical machine or a processor of a virtual machine. In the embodiment of the present application, the memory 520 stores at least one instruction, which is loaded and executed by the processor 510 to implement the methods in the above embodiments.

[0115] In an optional embodiment of the present application, the camera 530 may include a front camera or a rear camera.

[0116] Figure 6 A structural block diagram of a terminal 600 according to another embodiment of the present application is shown.

[0117] See also Figure 6 The terminal 600 includes: a processor 601 and a memory 602.

[0118] In the embodiment of the present application, the processor 601 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 601 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 601 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state.

[0119] The memory 602 may include one or more computer-readable storage media, which may be non-transitory. The memory 602 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments of the present application, the non-transitory computer-readable storage medium in the memory 602 is used to store at least one instruction, which is used to be executed by the processor 601 to implement the method in the embodiment of the present application.

[0120] In an optional embodiment of the present application, the terminal 600 further includes: a peripheral device interface 603 and at least one peripheral device. The processor 601, the memory 602 and the peripheral device interface 603 can be connected via a bus or a signal line. Each peripheral device can be connected to the peripheral device interface 603 via a bus, a signal line or a circuit board. Specifically, the peripheral device includes: at least one of a display screen 604, a camera 605 and a power supply 606.

[0121] The peripheral device interface 603 can be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 601 and the memory 602. In some embodiments of the present application, the processor 601, the memory 602, and the peripheral device interface 603 are integrated on the same chip or circuit board; in some other embodiments of the present application, any one or two of the processor 601, the memory 602, and the peripheral device interface 603 can be implemented on a separate chip or circuit board. This embodiment of the present application is not specifically limited to this.

[0122] The display screen 604 is used to display the UI (User Interface). The UI may include graphics, text, icons, videos and any combination thereof. When the display screen 604 is a touch display screen, the display screen 604 also has the ability to collect touch signals on the surface or above the surface of the display screen 604. The touch signal can be input to the processor 601 as a control signal for processing. At this time, the display screen 604 can also be used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments of the present application, the display screen 604 can be one, and the front panel of the terminal 600 is set; in other embodiments of the present application, the display screen 604 can be at least two, which are respectively set on different surfaces of the terminal 600 or are folded; in some other embodiments of the present application, the display screen 604 can be a flexible display screen, which is set on the curved surface or folded surface of the terminal 600. Even, the display screen 604 can also be set as a non-rectangular irregular shape, that is, a special-shaped screen. The display screen 604 can be made of materials such as LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode) and the like.

[0123] Camera 605 is used to capture images or videos. Optionally, camera 605 includes a front camera and a rear camera. Typically, the front camera is arranged on the front panel of the terminal, and the rear camera is arranged on the back of the terminal. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize the panoramic shooting and VR (Virtual Reality) shooting function or other fusion shooting functions. In some embodiments of the present application, camera 605 may also include a flash. The flash can be a monochrome temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation at different color temperatures.

[0124] The power supply 606 is used to power various components in the terminal 600. The power supply 606 can be an alternating current, a direct current, a disposable battery, or a rechargeable battery. When the power supply 606 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery charged through a wired line, and a wireless rechargeable battery is a battery charged through a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0125] The terminal structure block diagram shown in the embodiment of the present application does not constitute a limitation on the terminal 600. The terminal 600 may include more or fewer components than those shown in the figure, or combine certain components, or adopt a different component arrangement.

[0126] In this application, the terms "first", "second", etc. are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or order; the term "plurality" refers to two or more, unless otherwise expressly defined. The terms "installed", "connected", "connected", "fixed", etc. should be understood in a broad sense. For example, "connected" can be a fixed connection, a detachable connection, or an integral connection; "connected" can be a direct connection or an indirect connection through an intermediate medium. For those of ordinary skill in the art, the specific meanings of the above terms in this application can be understood according to the specific circumstances.

[0127] The above describes in detail the structure, features and effects of the present invention based on the embodiments shown in the drawings. The above is only a preferred embodiment of the present invention, but the present invention is not limited to the scope of implementation shown in the drawings. Any changes made according to the concept of the present invention, or modifications to equivalent embodiments with equivalent changes, which still do not exceed the spirit covered by the description and drawings, should be within the protection scope of the present invention.

Claims

1. A skin care effect detection method based on deep learning technology, characterized in that: include: S1: Obtain the initial skin image before care; S2: After preprocessing the initial skin image, perform matrix segmentation to generate N*N detection units; S3: A deep learning-based detection model built through local facial feature recognition and convolutional network learning, and trained with sample datasets; the sample datasets include wrinkle sample datasets, acne sample datasets, pore sample datasets, and spot sample datasets; S4: Use the detection model to detect each detection unit to determine the location of facial feature points and initial features; S5: obtaining a skin image after care; S6: Matching the treated skin image with the initial skin image, determining the corresponding position of the facial feature point on the treated skin image according to the position information of the facial feature point, and detecting the facial feature point on the treated skin image again using the detection model to obtain the post-treatment state; S7: Compare the initial features and post-care features of facial feature points and generate a care report; Step "S3: constructing a deep learning-based detection model through local facial feature recognition and convolutional network learning and training the detection model with a sample data set" specifically includes: S31: Collect frontal images with skin problems respectively, mark facial feature points with skin problems in the frontal images, and form a sample dataset D1 of facial feature points of problem skin; S32: Train the detection model, use the two-stage channel selection pre-training method, first train the backbone network ηmain from the ImageNet image classification dataset DCIs; then train the instance segmentation model mpretrained from the lesion boundary segmentation dataset DLBS published by ISIC; on the problem skin facial feature point sample dataset D1, select the best two channels of the instance segmentation model mpretrained to obtain mpretrained_2; then train to obtain the detection model ηdetect; Step "S4: using the detection model to detect each detection unit to determine the location of facial feature points and initial features" specifically includes: S41: predict m target detection frames for each detection unit, predict the horizontal coordinate x, vertical coordinate y, width w, height h of the target region of each target detection frame, and reliability information c of the target detection frame; S42: Using the detection model ηdetect to detect each target detection frame and obtain the location of the problematic skin feature point and initial feature data; The reliability information c is calculated as follows: C=P(o)*IOUt_p Where P(o) indicates whether the target detection frame contains the target area. If there is a target area, it is 1; if there is no target area, it is 0. IOUt_p represents the overlap between the predicted target area and the actual target area. If there is no target area in the target detection frame, c is 0, otherwise c represents the intersection ratio between the predicted target area and the actual target area. Step "S4: using the detection model to detect each detection unit to determine the location of facial feature points and initial features" also includes: S43: Collecting RGB color information of the detection unit, and evaluating the skin color of the detection unit area according to the color information; "S5: obtaining a skin care image after care" specifically includes: S51: Collect multiple face photos through a picture collector, including but not limited to frontal, left side, and right side photos; S52: synthesizing the multiple face photos collected above into a complete face image through a photo synthesis technology; Step "S6: matching the treated skin image with the initial skin image, determining the corresponding position of the facial feature point on the treated skin image according to the facial feature point position information, and again using the detection model to detect the facial feature point on the treated skin image to obtain the post-treatment state" includes the following steps: S61: Acquire a skin care image; S62: placing the treated skin image and the initial skin image in the same coordinate system; S63: Aligning the characteristic points on the treated skin image with the characteristic points on the initial skin image so as to match the treated skin image with the initial skin image; Step S42 may specifically include: S421: The detection model ηdetect is used to detect and segment the distribution map of the problem facial feature points and obtain the number of the problem facial feature points Ns, as well as the area Ai and position (xi, yi) of the i-th problem facial feature point; S422: Obtain the area Aj and position (xj, yj) of the facial feature point j adjacent to the i-th problematic facial feature point; S423: determining an initial feature based on the area and position relationship between the ith facial feature point and the adjacent jth facial feature point and determining the problem type of the problem skin feature point based on the initial feature; The above step "S423: determining the initial features according to the area and position relationship between the ith facial feature point and the adjacent jth facial feature point and determining the problem type of the problematic skin feature point according to the initial features" may include the following specific judgment process: S4231: If the ith facial feature point is linearly continuous with the adjacent jth facial feature point, and the length of the linear continuity of the adjacent facial feature points exceeds the first predetermined value K, it is indicated as a wrinkle and the number and length of the wrinkles are recorded; wherein the first predetermined value K is 3 mm; whether the two are adjacent can be determined by the coordinates of the positions of the problematic facial feature points; when the difference between the coordinates of the two is small, it indicates that the two are continuous; when the difference between the coordinates of the two is large, it indicates that there is a gap between the two; when the coordinates of several consecutive adjacent facial feature points are continuously increasing, it can be indicated as a wrinkle; S4232: If the ith facial feature point and the adjacent jth facial feature point are similar and discontinuous in a flake shape, and the RGB colors of the ith facial feature point and the jth facial feature point are in the range of yellow to dark brown, it indicates that there are spots in the area, and the size and color depth of the spots are recorded; S4233: If the ith facial feature point and the adjacent jth facial feature point are approximately circular, and the diameter of the circle is less than 0.5 mm, then the area is a pore; S4234: If the ith facial feature point and the adjacent jth facial feature point are continuous and approximately circular, and the diameter of the circle exceeds 1 mm, it is determined that the area is acne.

Images