System and method for monitoring human skin

Abstract

To monitor human skin, a camera and a depth sensor record a plurality of 3D surface images of the skin surface (S1). Each 3D surface image is taken from different distances and different angles with respect to the skin surface and includes a plurality of pixels with depth information and RGB color information. A computer system (1) detects skin surface features in the 3D surface images (S2) and uses the skin surface features and the depth information to determine the orientation and distance of each angle of the 3D surface image (S3). The computer system (1) uses the 3D surface images and their respective angular orientations and distances to generate a 6D model of the skin surface (S4). The 6D model of the skin surface includes a plurality of surface data points with 3D coordinates and RGB color information. The computer system (1) generates a dermatological evaluation result of the 6D model (S5) and draws the dermatological evaluation result on a user interface (S6).

Description

Technical Field

[0001] The present disclosure relates to a system and method for monitoring human skin. Specifically, the present disclosure relates to a computer system and a computer-implemented method for monitoring human skin.

Background Art

[0002] Telemedicine has become more common, thanks at least in part to the widespread availability and accessibility of the Internet. Telemedicine has become more acceptable even for medical consultations and initial diagnoses, especially during the COVID-19 pandemic or for patients in remote locations, where it has become almost essential. This is also true in the case of dermatology, where online websites can be used to receive photos of a patient's skin from the patient for diagnosis and treatment recommendations. It seems very easy for patients to record and upload such photos using their mobile phones and / or personal computers. This type of telemedicine seems very practical, as it makes it much easier for patients to address skin conditions or symptoms without having to leave their homes. Nevertheless, while remote dermatological diagnosis based on photos can be suitable and may be accurate in some cases, it is not considered highly accurate due to the limitations of reliability and the information content of the photos.

Summary of the Invention

[0003] The object of the present disclosure is to provide a system and method for monitoring human skin. Specifically, the object of the present disclosure is to provide a computer-implemented method and a computer system for monitoring the human skin system, which method and system do not have at least some of the drawbacks of the prior art.

[0004] According to the present disclosure, these objects are addressed by the features of the independent claims. Additionally, further advantageous embodiments can be obtained from the dependent claims and the present description.

[0005] According to the present disclosure, the above objects are in particular realized in that a computer-implemented method for monitoring human skin comprises a camera and a depth sensor for recording a plurality of 3D surface images of the skin surface. Each 3D surface image is taken from different angles and different distances with respect to the skin surface and comprises a plurality of pixels with depth information and RGB color information (3D = three-dimensional; RGB = red, green, blue). The computer system detects skin surface features in the 3D surface images. The computer system uses the skin surface features and the depth information to determine the orientation and distance of each angle of the 3D surface images. The computer system uses the 3D surface images as well as their respective angular orientations and distances to generate a 6D model of the skin surface (6D = six-dimensional). The 6D model of the skin surface comprises a plurality of surface data points. Each surface data point comprises 3D coordinates and RGB color information. The computer system generates a dermatological evaluation result of the 6D model of the skin surface and renders the dermatological evaluation result on a user interface. For example, the plurality of 3D surface images of the skin surface are recorded using a camera and a depth sensor of a mobile electronic device, such as a handheld electronic device, such as a mobile communication device, such as a mobile phone (smartphone or smartwatch) or a tablet computer.

[0006] In one embodiment, the computer system determines a body part from the 3D surface image to detect skin surface features in the 3D surface image and uses an initial coordinate system and respective feature contours associated with the body part.

[0007] In one embodiment, the computer system selects a 3D reference image from the plurality of 3D surface images. The 3D reference image has a coordinate system with an angular orientation closest to the initial coordinate system by comparison. The computer system uses the 3D reference image to generate a 6D model of the skin surface.

[0008] In one embodiment, the computer system starts with a 3D reference image and generates a 6D model of the skin surface by determining adjacent 3D surface images such that two adjacent 3D surface images have orientations and distances that are closest to each other by comparison, and starting from the 3D surface image adjacent to the 3D reference image, rotates and translates the adjacent 3D reference images towards the 3D reference image to match their respective angular orientations and distances.

[0009] In one embodiment, the computer system includes in the dermatological evaluation results a probabilistic ranking of dermatological diagnoses related to skin diseases, skin problems, and / or skin types. For example, the dermatological evaluation results include a list of skin diseases, skin problems, and / or skin types, and a probabilistic ranking of these skin diseases, skin problems, and / or skin types.

[0010] In one embodiment, the computer system uses the 6D model of the skin surface as input to a neural network to generate dermatological evaluation results. The neural network is trained to identify skin diseases, skin problems, and / or skin types using multiple 6D models of the skin surface from multiple people. The 6D model of the skin surface includes a plurality of surface data points with 3D coordinates and RGB color information.

[0011] In one embodiment, the computer system generates a 5D map of the skin surface (5D = five - dimensional) by applying a projection onto the 6D model of the skin surface. The 5D map of the skin surface includes a plurality of map data points. Each map data point includes 2D coordinates and RGB color information (2D = two - dimensional). The computer system uses the 5D map of the skin surface as input to a neural network to generate dermatological evaluation results. The neural network is trained to identify skin diseases, skin problems, and / or skin types using multiple 5D maps of the skin surface from multiple people. The 5D map of the skin surface includes a plurality of map data points with 2D coordinates and RGB color information.

[0012] In one embodiment, the computer system generates one or more 6D submodels of the skin surface by extracting one or more regions of the 6D model of the skin surface. Each of the 6D submodels of the skin surface comprises a plurality of surface data points of the respective region. Each surface data point comprises 3D coordinates and RGB color information. The computer system uses one or more 6D submodels of the skin surface as input to a neural network to generate a dermatological evaluation result. The neural network is trained to identify skin diseases, skin problems, and / or skin types using a plurality of 6D submodels of the skin surface from a plurality of people. One or more 6D submodels of the skin surface comprise a plurality of surface data points of the respective region with 3D coordinates and RGB color information.

[0013] In one embodiment, the computer system generates one or more 6D submodels of the skin surface and / or a 5D map of the skin surface. The computer system generates one or more 6D submodels of the skin surface by extracting one or more regions of the 6D model of the skin surface. Each of the 6D submodels of the skin surface comprises a plurality of surface data points of the respective region. Each surface data point comprises 3D coordinates and RGB color information. The computer system generates a 5D map of the skin surface by applying a projection onto the 6D model of the skin surface. The 5D map of the skin surface comprises a plurality of map data points. Each map data point comprises 2D coordinates and RGB color information. The computer system uses the 6D model of the skin surface, one or more 6D submodels of the skin surface, and / or the 5D map of the skin surface as input to a neural network to generate a dermatological evaluation result. The neural network is trained to identify skin diseases, skin problems, and / or skin types using a plurality of 6D models of the skin surface from a plurality of people, a plurality of one or more 6D submodels of the skin surface from a plurality of people, and / or a plurality of 5D maps of the skin surface from a plurality of people.

[0014] In one embodiment, the computer system uses a 6D model of the skin surface to determine skin surface characteristics. The skin surface characteristics include wrinkles, wrinkle dimensions, pores, pore dimensions, sebaceous cysts, nodules, nodule dimensions, nodule shapes, macules, macule dimensions, macule shapes, papules, papule dimensions, papule shapes, spots, spot dimensions, spot shapes, pustules, pustule dimensions, blisters, blister dimensions, eruptions, eruption dimensions, eruption shapes, rashes, erosions, erosion dimensions, erosion shapes, ulcers, ulcer dimensions, ulcer shapes, crusts, scales, scale types, skin cracks, and / or atrophy. The computer system uses the skin surface characteristics to generate a dermatological evaluation result.

[0015] In one embodiment, when the computer system completes a first dermatological evaluation for a first 6D model of the skin surface, it stores the first 6D model of the skin surface and the first dermatological evaluation result in the data storage. When the computer system receives a second 6D model of the skin surface, it identifies the first dermatological evaluation result in the data storage by comparing the second 6D model of the skin surface with the first 6D model of the skin surface. When the computer system completes a second dermatological evaluation for the second 6D model of the skin surface, it generates a tracking report by comparing the second dermatological evaluation result with the first dermatological evaluation result, and draws the tracking report on the user interface.

[0016] In one embodiment, the computer system draws a 6D model of the skin surface on a display device, receives evaluation data from the user via the user interface, and uses the evaluation data to generate a dermatological evaluation result.

[0017] In one embodiment, the electronic device of the computer system transmits a 6D model of the skin surface to the processing system of the computer system via a communication network. The processing system uses the 6D model of the skin surface received from the electronic device to generate a dermatological evaluation result. The processing system transmits the dermatological evaluation result to the electronic device.

[0018] In addition to the method of monitoring human skin, the present disclosure also relates to a computer system. The computer system comprises a camera, a depth sensor, a user interface, and one or more processors. The one or more processors are configured to control the camera and the depth sensor to record a plurality of 3D surface images of the skin surface. Each 3D surface image is taken from different angles and different distances with respect to the skin surface and comprises a plurality of pixels with depth information and RGB color information. The one or more processors are configured to detect skin surface features in the 3D surface images. The one or more processors are configured to use the skin surface features and the depth information to determine the orientation and distance of each angle of the 3D surface image. The one or more processors are configured to use the 3D surface images and their respective orientations and distances of each angle to generate a 6D model of the skin surface. The 6D model comprises a plurality of surface data points, and each surface data point comprises 3D coordinates and RGB color information. The one or more processors are configured to generate a dermatological evaluation result of the 6D model of the skin surface and to draw the dermatological evaluation result on the user interface.

[0019] The one or more processors are configured to execute the above method of monitoring human skin.

[0020] In one embodiment, a computer system comprises an electronic device and a processing system. The electronic device includes one or more processors configured to transmit a 6D model of the skin surface to the processing system via a communication network. The processing system includes one or more processors configured to use the 6D model of the skin surface received from the electronic device to generate a dermatological evaluation result and to transmit the dermatological evaluation to the electronic device. For example, the electronic device is a mobile electronic device, such as a handheld electronic device, such as a mobile communication device, such as a mobile phone (smartphone or smartwatch) or a tablet computer. For example, the computerized processing system is a computerized server system, such as a computer system using a cloud.

[0021] In addition to the method and computer system for monitoring human skin, the present disclosure also relates to a computer program product comprising computer program code for controlling a processor of an electronic device. Specifically, the present disclosure relates to a computer program product comprising a computer-readable medium having stored thereon the computer program code, for example, a non-transitory computer-readable medium. The computer program code is configured to control a processor of an electronic device, the electronic device comprising a camera, a depth sensor, and a user interface connected to the processor, and this control is performed such that the processor controls the camera and the depth sensor to record a plurality of 3D surface images of the skin surface of human skin. Each 3D surface image is taken from different angles and different distances with respect to the skin surface and comprises a plurality of pixels accompanied by depth information and RGB color information. The computer program code is further configured to control a processor of the electronic device, and this control is performed such that the processor detects skin surface features in the 3D surface images; determines the orientation and distance of each angle of the 3D surface images using the skin surface features and the depth information; generates a 6D model of the skin surface using the 3D surface images and their respective orientation and distance of each angle, whereby the 6D model of the skin surface comprises a plurality of surface data points, and each surface data point comprises 3D coordinates and RGB color information; transmits the 6D model of the skin surface to a computerized processing system; receives a dermatological evaluation result of the 6D model of the skin surface from the computerized processing system; and renders the dermatological evaluation result on the user interface. For example, the electronic device is a mobile electronic device, such as a handheld electronic device, such as a mobile communication device, such as a mobile phone (smartphone or smartwatch) or a tablet computer.

[0022] The computer program code is further configured to control a processor of the electronic device to execute the above method for monitoring human skin.

[0023] The present disclosure also relates to a computer program product comprising computer program code for controlling one or more processors of a computerized processing system, in addition to a method and a computer system for monitoring human skin. Specifically, the present disclosure relates to a computer program product comprising a computer-readable medium having stored thereon computer program code, such as a non-transitory computer-readable medium. The computer program code is configured to control one or more processors of a computerized processing system to receive a 6D model of a human skin surface from an electronic device, whereby the 6D model of the skin surface comprises a plurality of surface data points, and each surface data point comprises 3D coordinates and RGB color information. The computer program code is further configured to control one or more processors of the computerized processing system to generate a dermatological evaluation result of the 6D model of the human skin surface and to transmit the dermatological evaluation result to the electronic device. For example, the computerized processing system is a computerized server system, such as a computer system using a cloud.

[0024] The computer program code is further configured to control one or more processors of a computerized processing system to execute the above method for monitoring human skin.

[0025] The present disclosure will be described in more detail by way of example with reference to the drawings, in which:

Brief Description of the Drawings

[0026]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

DETAILED DESCRIPTION OF THE INVENTION

[0027] In FIGS. 1 to 4, reference numeral 1 refers to a computer system. The computer system 1 includes one or more processors 100, 100* arranged in one or more devices according to embodiments and configurations. FIGS. 1 and 3 illustrate an embodiment in which one or more processors 100 of the computer system 1 are arranged in one electronic device 10. FIGS. 2 and 4 illustrate an embodiment in which one or more processors 100, 100* of the computer system 1 are arranged in two or more devices, particularly in an electronic device 10 and a computerized processing system 11.

[0028] The electronic device 10 is a personal computing device, such as a personal computer, such as a laptop or desktop computer, or a mobile computing device, such as a handheld device, such as a mobile phone (smartphone or smartwatch) or a tablet computer. As schematically illustrated in FIGS. 1, 2, and 8, the computer system 1 comprises a digital camera 101, a depth sensor 102, and a user interface 103. In the embodiments illustrated in FIGS. 1, 2, and 8, the electronic device 10 is schematically shown to include a digital camera 101, a depth sensor 102, and a user interface 103. Nevertheless, those skilled in the art will understand that the components of the electronic device 10 can be arranged in separate enclosures and connected to the processor 100 of the electronic device 10 via wireless or wired communication links; in particular, the digital camera 101 and the depth sensor 102 can be arranged in a separate enclosure or unit, such as a handheld camera unit, and / or the user interface 103 can be arranged in a separate unit, such as a screen terminal and / or a keyboard, a computer mouse, a touchpad, and the like. The camera 101, the depth sensor 102, and the user interface 103 are connected to the processor 100. The user interface 103 comprises a display device, a keyboard, a touch-sensitive display device, a microphone, and / or a speaker. The depth sensor 102 comprises an infrared, LIDAR (Light Detection and Ranging), ultrasonic, or any other suitable depth or distance measurement sensor. Modern mobile phones have an integrated combination of a digital camera and a depth sensor. For example, the depth sensor 102 has a depth resolution of at least 1 mm to 0.1 mm or better. The processor 100 is configured to control the digital camera 101 and the depth sensor 102 to capture a 3D surface image with RGB color (red, green, blue) information, for example 8-bit to 16-bit color information per color. The 3D surface image is defined in a local coordinate system, specifically in camera coordinates based on the camera coordinate system.The definition of the 3D surface image in the local coordinate system includes camera-specific parameters such as focal length, focus, and not only the rotation of the angle with respect to the earth's gravity, angular acceleration and angular velocity, but also device-specific data such as lateral acceleration and velocity. Pixel data, together with depth data and camera-specific parameters (e.g., restricted length), is used to calculate the 3D surface image. The 3D surface image has a full HD (high definition) or higher (reaching at least 1920×1080 = 2073600 pixels) 2D resolution, and a downsampled resolution of 640 by 480 pixels (640×480 = 307200 pixels). The depth or distance measurement has an accuracy or resolution better than or equal to 1 mm to 0.1 mm, and a capture resolution of the depth scale of at least 640 by 480 (=307200). Those skilled in the art will understand that in order to obtain the defined target resolution of the 6D model of the skin surface, which is described and defined in more detail below, a lower resolution of the digital camera 101 and / or the depth sensor 102 may be adopted when increasing the total number of 3D surface images to be captured. As schematically illustrated in FIGS. 1, 2, and 8, the digital camera 101 and the depth sensor 102 are configured to capture and record a 3D surface image with RGB color information of the skin surface 20 of the human body part 2, and in the examples of FIGS. 1, 2, and 8, the facial skin surface 20 of the human face 2. As illustrated in FIG. 8, the 3D surface image is captured from different positions P1, P2 having different angles α1 / β1, α2 / β2 and different distances d1, d2 with respect to the skin surface 20, and more specifically, with respect to the reference coordinate system C(x, y, z) and the reference point R of the skin surface 20, which will be described in more detail below.

[0029] The computerized processing system 11 comprises one or more computers having one or more processors 100*, for example, a computerized server.

[0030] As illustrated in FIG. 2, the electronic device 10 and the computerized processing system 11 are connected via the communication network 2. The electronic device 10 and the computerized processing system 11 are configured to perform data communication via the communication network 2. The communication network includes a local area network (LAN), a wireless local area network (WLAN), a mobile wireless network such as a GSM (Global System for Mobile Communication) or UMTS (Universal Mobile Telephone System) network, and / or the Internet.

[0031] In the following paragraphs, with reference to FIGS. 3 to 8, described is a possible order of steps to be executed by the computer system 1 for monitoring human skin, or by one or more of its processors 100, 100*.

[0032] As illustrated in FIGS. 3 and 4, in step S1, the computer system 1 captures and records a 3D surface image with RGB color information of the skin surface 20 of the human body part 2. More specifically, the computer system 1 or its processor 100 controls the digital camera 101 and the depth sensor 102 respectively to capture and store a 3D surface image with RGB color information.

[0033] In step S1*, the positions of the digital camera 101 and the depth sensor 102 are changed based on the skin surface 20 or the body part 2. Either the positions of the camera 101 and the depth sensor 102 change, or the body part 2 moves. In one embodiment, the computer system 1 or its processor 100 respectively generates and draws instructions on the user interface 103 regarding how to change the respective positions, visually, for example, on a display device and / or acoustically via a speaker. Alternatively, the computer system 1 or its processor 100 respectively controls an electric motor to change, for example, rotate the digital camera 101 and the depth sensor 102 to different respective positions.

[0034] Steps S1 and S1* are repeated to capture a plurality of 3D surface images with RGB color information of the skin surface 20 of the human body part 2 from different positions P1, P2 having different angles α1 / β1, α2 / β2 and different distances d1, d2 with respect to the skin surface 20. For example, the plurality of 3D surface images are captured by the user moving the camera 101 and the depth sensor 102 of the handheld electronic device 10, and this movement is a continuous movement taking a distance of 10 to 30 centimeters over the area of the skin surface 20 that is to be monitored, for example, at the front of the face or a specific area of the face. One skilled in the art will understand that in order to capture a larger skin area, for example, a full body image, the distance can be increased, for example, up to 150 cm. For example, a plurality of approximately 30 3D surface images are captured per second during a recording time of approximately 5 to 25 seconds, and the captured and stored 3D surface images reach approximately 150 to 750. Nevertheless, one skilled in the art will understand that a smaller or larger number of 3D surface images, for example, 1 to 60, can be captured during a shorter or longer recording time, for example, 1 to 90 seconds respectively.

[0035] As schematically illustrated in FIGS. 3 and 4, at an optional step S2*, the computer system 1 or its processor 100 each determines a body part 2. In certain embodiments, the body part 2 is selected by the user via the user interface 103 prior to step S1. In another embodiment, the body part 2 is determined from the first 3D surface image by the computer system 1 or its processor 100 each, for example as part of step S1. The computer system 1 or its processor 100 each determines, as schematically illustrated in FIGS. 7 and 8, an initial coordinate system C, e.g., a 3D xyz coordinate system C having a reference point R, and characteristic contours F, F1, F2, F3 of skin surface features (a “landmark”) associated with the body part 2 and its skin surface 20. The characteristic contours F, F1, F2, F3 are three-dimensional (3D) and determine the location and approximate area or shape of characteristic features of each body part 2 in the initial coordinate system C. For example, in the case of a human face, the characteristic contours define the contour F of the face, the contours F1 of both eyes, the contour F2 of the nose, and the contour F3 of the mouth, as illustrated in FIGS. 7 and 8. In the case of a human foot, the characteristic contours define the contours of the toes and / or toenails of the foot. In the case of a human hand, the characteristic contours define the contours of the fingers and / or fingernails of the hand. In a further embodiment, the computer system 1 or its processor 100 each receives from the user via the user interface 103 a command and / or a specification of at least a part of the initial coordinate system C. For example, if the back is the determined / selected body part 2, the computer system 1 or its processor 100 each receives from the user via the user interface 103 a user command and / or a specification defining the selected area of the back to be captured.

[0036] In step S2, the computer system 1 or its processor 100 each detect skin surface features ( "landmarks") in the recorded 3D surface image of the skin surface 20. More specifically, the computer system 1 or its processor 100 each detect skin surface features using the feature contour lines F, F1, F2, F3 associated with the determined body part 2. The skin surface features are defined in a local coordinate system, specifically in camera coordinates based on the camera coordinate system. In the case of the face, the computer system 1 or its processor 100 each detect the contour line F of the face, the contour lines F1 of both eyes, the contour line F2 of the nose, and / or the contour line F3 of the mouth. In one embodiment, the computer system 1 or its processor 100 each superimpose the detected skin surface features on the image captured by the digital camera 101 and shown on the display device 103 of the user interface, as shown in FIG. 8.

[0037] In step S3, the computer system 1 or its processor 100 respectively determines the orientation angle and distance of the captured 3D surface image of the skin surface 20. More specifically, the computer system 1 or its processor 100 respectively uses the detected skin surface features and depth information of the 3D surface image to determine the orientation angle and distance of the captured 3D surface image. The orientation angle and distance of the captured 3D surface image are defined based on a global coordinate system, more specifically, its local coordinate system based on the reference coordinate system C(x, y, z). For example, the orientation angle and distance of the captured 3D surface image are respectively defined by the angles α1 / β1 (or α2 / β2) and the distance d1 (or d2) with respect to the reference coordinate system C(x, y, z) and the reference point R of the skin surface 20, as schematically illustrated in FIG. 8. The computer system 1 or its processor 100 respectively stores the captured 3D surface images of the skin surface 20, as well as their respective angles α1 / β1, α2 / β2, and distances d1, d2 in the local memory. Once the computer system 1 or its processor 100 respectively determines that a sufficient number of 3D surface images have been captured to generate a 6D model of the skin surface 20 of the body part 2, the process continues in step S4. For example, if the number of captured 3D surface images exceeds the minimum threshold of resolution, for example, the minimum number of 3D surface images from the minimum number of different positions P1, P2, it is sufficient to guarantee a sufficient target range and resolution of the 6D model of the skin surface 20 (see also the respective explanations related to step S5). Otherwise, the process continues in step S1* by changing the respective positioning of the digital camera 101 and the skin surface 20 of the body part 2 to capture another 3D surface image from different positions P1, P2 in step S1. For example, the target resolution for the 6D model of the skin surface 20 is respectively at least 1 mm, that is, one surface data point per 1 mm 2 or the target resolution for the 6D model of the skin surface 20 is even higher, for example, at least 0.1 mm, that is, one surface data point per 1 mm 2There are 100 surface data points per unit area. In the case of a human face, a surface area of approximately 20 cm × 25 cm is captured, each requiring at least 500,000 or 5,000,000 surface data points.

[0038] In step S4, the computer system 1 or its processor 100 each generates a 6D model of the skin surface 20. More specifically, the computer system 1 or its processor 100 each uses the captured and stored 3D surface images and their respective angular orientations (α1 / β1, α2 / β2) and distances (d1, d2) to generate a 6D model of the skin surface 20. The 6D model of the skin surface 20 is defined with reference to a global coordinate system, more specifically the reference coordinate system C(x, y, z). The 6D model of the skin surface 20 comprises a plurality of surface data points. Each surface data point of the 6D model has 3D coordinates [x1, y1, z1…x n ,y n ,z n and RGB color information [r1, g1, b1…r n ,g n ,b n and is represented as follows, as exemplified in matrix K below, for the 6D model of the skin surface 20: [Number]

[0039] To generate the 6D model K of the skin surface 20, the computer system 1 or its processor 100 each selects a 3D reference image from the captured and stored 3D surface images. The 3D reference image has an orientation at an angle closest to the initial coordinate system C. In other words, the 3D reference image is a 3D surface image captured using the optical axis of a digital camera aligned closest to the z-axis of the initial coordinate system C and corresponds to the top view schematically illustrated in FIG. 8. The 3D reference image is not transformed and remains unchanged. The computer system 1 or its processor 100 each uses the selected 3D reference image to generate the 6D model K of the skin surface 20. More specifically, starting from the 3D reference image, the computer system 1 or its processor 100 each determines adjacent 3D surface images from the captured and stored 3D surface images. Two adjacent (or neighboring) 3D surface images have orientations and distances at the closest angles with respect to each other. To generate the 6D model K of the skin surface 20, the computer system 1 or its processor 100 each rotates and translates (moves) the adjacent 3D reference images toward the 3D reference image. Thus, starting from the 3D surface image adjacent to the 3D reference image, the adjacent 3D surface images are rotated to align with the orientation of the 3D reference image and translated (moved) to the same distance as the 3D reference image. To generate the 6D model K of the skin surface 20, the computer system 1 or its processor 100 each determines the rotation and / or translation transformation by optimizing the degree of coincidence of the 3D data points (x, y, z) of the skin surface features, also referred to as "3D landmarks," of the two adjacent 3D surface images being processed. For example, the transformation is optimized with respect to the best fit or minimum error measure. The computer system 1 or its processor 100 each stores the surface data points for the 6D model K of the skin surface 20 when the degree of coincidence of the superimposed 3D landmark data is within the range of the optimization threshold value, for example, within the range of the best fit or error threshold value.

[0040] As illustrated in FIGS. 3, 4, 5, and 6, in step S5, the computer system 1 generates a dermatological evaluation result using the 6D model K of the skin surface 20. After processing all the captured and recorded 3D surface images, if the resulting 6D model K of the skin surface 20 is insufficient with respect to the desired quality, for example, the resolution or number of surface data points, the computer system 1 or its processor 100 will each instruct the user to proceed to steps S1* and S1 to capture and record additional 3D surface images, as schematically shown by step S40 in FIGS. 3 and 4.

[0041] As schematically illustrated in FIG. 3, steps S1, S1*, S2, S2*, S3, S4, S5, and S6 are executed by the computer system 1. In certain embodiments, step S5 is executed by the electronic device 10 of the computer system 1 or its processor 100, respectively.

[0042] As schematically illustrated in the embodiment or the configuration of FIG. 4, steps S1, S1*, S2, S2*, S3, S4, and S6 are executed by the electronic device 10 of the computer system 1 or its processor 100, respectively. In step S7, the 6D model K of the skin surface 20 is transmitted from the electronic device 10 to the computerized processing system 11 via the communication network 2. Step S5 is executed by the computerized processing system 11 of the computer system 1 or its processor 100*, respectively.

[0043] As illustrated in FIG. 5, in different embodiments, step S5 further comprises the computer system 1 that executes optional step S51 to generate one or more 6D sub - models of the skin surface 20 and / or optional step S52 to generate a 5D map of the skin surface 20.

[0044] In step S51, the computer system 1 generates one or more 6D submodels of the skin surface 20. Essentially, the 6D submodel of the skin surface 20 is an extracted area of the 6D model of the skin surface 20. Each 6D submodel or extracted area is defined, for example, by the points of the central area and the area radius, by the specific skin surface feature contours F1, F2, F3, by the defined 2D area contour, and / or by the defined 3D area contour. The 6D submodel of the skin surface 20 comprises a plurality of surface data points for each area, and each surface data point comprises 3D coordinates and RGB color information. In the case of a human face, examples of 6D submodels or extracted areas include eyes, nose, mouth, cheeks, forehead, or parts thereof.

[0045] Each surface data point of the 6D submodel has 3D coordinates [x u , y u , z u … x v , y v , z v and RGB color information [r u , g u , b u … r v , g v , b v and represents the 6D submodel of the skin surface 20 as exemplified in matrix S below:

Number

[0046] In step S52, the computer system 1 generates a 5D map of the skin surface 20. More specifically, the computer system 1 generates a 5D map 20 of the skin surface by applying a projection to the 6D model of the skin surface 20. For example, the projection is performed along the z-axis of the reference coordinate system C or the reference image. Other projection axes are possible. The 5D map of the skin surface 20 comprises a plurality of map data points. Each map data point comprises 2D coordinates and RGB color information.

[0047] Each map data point of the 5D map has 2D coordinates [x u , y u … x v , y v and RGB color information [r u , g u , b u … r v , g v , b v and represents the 5D map of the skin surface 20 as exemplified in matrix M below:

Number

[0048] As schematically illustrated in FIG. 5, the computer system 1 generates a dermatological evaluation result by executing a neural network in step S54, by executing various filtering algorithms in step S55, and / or by rendering a 6D model of the skin surface 20 on a display device and receiving evaluation data from the user via the user interfaces 103, 104.

[0049] The following table summarizes various possible embodiments and / or configurations K, S, and / or M of the input data, as well as steps S54, S55, S56 for generating a dermatological evaluation:

Table 1

[0050] In step S54, the computer system 1 executes a neural network. For example, the neural network is a multi-label deep feed-forward neural network, a convolutional neural network, or a transformer neural network. The neural network is trained to identify skin diseases, skin problems, and / or skin types using a plurality of 6D models K of the skin surface 20 from a plurality of people, a plurality of 6D sub-models S of the skin surface 20 from a plurality of people, and / or a plurality of 5D maps M of the skin surface 20 from a plurality of people. Therefore, the computer system 1 executes the neural network using the 6D model K of the skin surface 20 generated in step S4, one or more 6D sub-models S of the skin surface 20 generated in step S51, and / or the 5D map M of the skin surface 20 generated in step S52. In one embodiment, the neural network is trained to identify skin diseases, skin problems, and / or skin types along with the associated probabilities. For example, the neural network uses a tree structure that labels skin diseases, skin problems, and / or skin types as illustrated in FIG. 9. In a further embodiment, the neural network is trained and executed to detect skin surface characteristics as described below in relation to step S55.

[0051] In step S55, the computer system 1 executes various filter algorithms to determine the characteristics of the skin surface, particularly the skin surface characteristics related to specific skin diseases, skin problems, and / or skin types. The computer system 1 determines the skin surface characteristics by applying various filter algorithms to the 6D model K of the skin surface 20 generated in step S4, to one or more 6D sub-models S of the skin surface 20 generated in step S51, and / or to the 5D map M of the skin surface 20 generated in step S52. For example, the skin surface characteristics include wrinkles, wrinkle dimensions, pores, pore dimensions, milia, nodules, nodule dimensions, nodule shapes, maculas, macula dimensions, macula shapes, papules, papule dimensions, papule shapes, pustules, pustule dimensions, blisters, blister dimensions, wheals, wheal dimensions, wheal shapes, acne, erosions, erosion dimensions, erosion shapes, ulcers, ulcer dimensions, ulcer shapes, crusts, scales, scale types, skin cracks, and / or atrophy. The filter algorithms include algorithms for determining the depth gradient (first derivative) at each surface data point of the 6D model or 6D sub-model, for determining the curvature scale (second derivative) at each surface data point of the 6D model or 6D sub-model, for determining the color spectrum scale of the region data of the 6D model or 6D sub-model, for determining statistical scales of gradient and curvature data such as mean and / or standard deviation, for determining density by measuring regions with changes in gradient and curvature, and / or for combined scales of surface change selection and color histogram, etc.

[0052] For example, the computer system 1 applies a linear-type filter, for example, a gradient calculation that shows the rising change from one pixel point to its geometric neighborhood. Numerical differentiation is calculated in all three dimensions by finite discretization. By calculating statistical measures, such as the mean or variance, the average change or the variance of the change is determined for a specific region, such as the cheek or eye region. Different values for the average gradient and the variance of the gradient determine different skin surface characteristics. To identify more complex skin surface characteristics associated with a specific disease, such as squamous cell carcinoma, more complex filters are used. For example, these filters are configured to detect rings in the 5D map of the skin surface 20, or dome-shaped structures associated with carcinomas in a 6D model or 6D submodel of the skin surface. For 2D detection, the filter includes an edge detection filter, which, in combination with a structure filter that determines the correlation with a circular structure, detects significant changes in the gradient, i.e., those that exceed the gradient threshold. The same approach is done in 3D by calculating the local gradient and detecting points with significant changes. To detect skin surface characteristics with more complex geometric forms, the above neural network is trained and used as appropriate.

[0053] In step S56, the computer system 1 draws a 6D model of the skin surface 20 on the display device. The drawing on the display device is adjusted, for example, in response to user navigation commands such as left rotation, right rotation, up rotation, down rotation, zoom in, zoom out, etc. Based on the visual analysis of the 6D model of the skin surface 20, the computer system 1 obtains evaluation data from the user via the user interfaces 103, 104. In one embodiment, the user evaluation data responds to and / or is complementary to the dermatological evaluation data generated by the computer system 1.

[0054] Furthermore, in step S5, the computer system 1 generates a dermatological evaluation result of the 6D model of the skin surface 20. More specifically, the computer system 1 uses the skin disease, skin problem, and / or skin type identified in step S54, the related probability determined in step S54, the skin surface characteristics determined in step S54 and / or S55, and / or the user evaluation data received in step S56 to generate a dermatological evaluation result of the 6D model of the skin surface 20. The dermatological evaluation identifies skin diseases, skin problems, and / or skin types. In certain embodiments, the dermatological evaluation identifies several skin diseases, skin problems, and / or skin types along with the related probabilities. The computer system 1 stores the dermatological evaluation result generated in step S5. In certain embodiments, the dermatological evaluation result further includes the suitability of therapies and / or skin care and / or treatment products related to the listed skin diseases, skin problems, and / or skin types, which are retrieved from a database and included in the computer system 1.

[0055] In response to the acquisition or receipt of the 6D model of the skin surface 20 for evaluation, in the embodiment or configuration of FIG. 6, in step S58, the computer system 1 determines whether a related dermatological evaluation result has been previously stored for a specific individual. In certain embodiments, the specific individual is identified by the computer system 1 using the acquired or received 6D model of the skin surface 20 to be evaluated. More specifically, the computer system 1 compares the acquired or received 6D model with the 6D models of the skin surface 20 that have been previously evaluated and stored. The degree of coincidence between the previously evaluated and stored 6D model and the acquired or received 6D model is determined by the computer system 1 based on a similarity measure that exceeds a defined similarity threshold. The similarity measure is based on defined criteria, for example, biometric characteristics determined from the acquired and stored 6D models of the skin surface.

[0056] As illustrated in FIG. 6, when the computer system 1 identifies relevant dermatological evaluation results that have been previously generated and stored for a specific individual, the process continues in step S59.

[0057] In step S59, the computer system compares the current dermatological evaluation result generated in step S5 and stored in step S58 with the relevant dermatological evaluation results previously generated and stored for the specific individual.

[0058] In step S60, in response to the comparison in step S59, the computer system 1 generates a tracking report. The tracking report displays the changes identified by comparing the current dermatological evaluation result with the relevant previously generated dermatological evaluation results for the specific individual. The changes included in the tracking report include positive changes, i.e., improvements in dermatological diseases and problems, and / or negative changes, i.e., exacerbations and / or new detections of dermatological diseases and problems. The computer system 1 includes the tracking report in the dermatological evaluation result.

[0059] In step S6, the computer system 1 draws the dermatological evaluation result generated in step S5 on the user interface 103. In some embodiments, step S6 is executed by the electronic device 10 of the computer system 1 or its processor 100, respectively. As schematically illustrated in the embodiment or the configuration of FIG. 4, in step S8, the dermatological evaluation result is transmitted from the computerized processing system 11 to the electronic device 10 via the communication network 2.

[0060] Note that the disclosed computer-implemented method and related computer system 1 for monitoring human skin provides not only an assessment of the type of human skin, but also a measurement-based identification of skin diseases, skin problems, and / or skin types. The disclosed method and computer system 1 enables tracking of the performance (over time) of skin care products for medical and / or cosmetic purposes. The disclosed method and computer system 1 can also be applied in telemedicine to the remote recording and remote findings of a human's skin, particularly the skin of the human face, by a medically trained professional. The 6D model of the skin surface 20 can be displayed photorealistically in 3D, and a tool for evaluating the skin by zooming, rotating, and displaying the 6D model of the skin surface 20 is provided to the professional.

[0061] It should be further noted that, although the steps have been presented in a particular order herein, those skilled in the art will understand that at least some of the order of the steps can be changed without departing from the scope of the disclosure.

Claims

1. Recording (S1) 3D surface images of a plurality of skin surfaces (20) by means of a camera (101) and a depth sensor (102), each 3D surface image being taken from different angles (α1 / β1, α2 / β2) and different distances (d1, d2) with respect to the skin surface (20) and comprising a plurality of pixels with depth information and RGB color information; Detecting (S2) skin surface features (F1, F2, F3) in the 3D surface images by means of a computer system (1); Using the skin surface features (F1, F2, F3) and the depth information to determine (S3) the orientation and distance of each angle of the 3D surface images by means of a computer system (1); Using the 3D surface images and their respective orientations and distances of each angle to generate (S4) a 6D model of the skin surface (20) by means of a computer system (1), the 6D model of the skin surface (20) comprising a plurality of surface data points, each surface data point comprising 3D coordinates and RGB color information; Generating (S5) a dermatological evaluation result of the 6D model of the skin surface (20) by means of a computer system (1); Rendering (S6) the dermatological evaluation result on a user interface (103) by means of a computer system (1) A computer-implemented method for monitoring human skin, comprising the steps of:

2. The method according to claim 1, wherein the computer system (1) for detecting skin surface features (F1, F2, F3) in the 3D surface images uses an initial coordinate system (C) associated with the body part (2) and respective characteristic contour lines (F1, F2, F3) determined from the 3D surface images to detect the skin surface features (F1, F2, F3).

3. The method according to claim 2, further comprising a computer system (1) that selects a 3D reference image having a coordinate system with an orientation of the angle closest to the initial coordinate system (C) by comparison from the plurality of 3D surface images and uses the 3D reference image to generate (S4) a 6D model of the skin surface (20).

4. Starting from the 3D reference image, adjacent 3D surface images are determined such that two adjacent 3D surface images have orientations and distances that are closest to each other by comparison, and starting from the 3D surface image adjacent to the 3D reference image, the adjacent 3D reference image is rotated and translated towards the 3D reference image to match their respective angular orientations and distances, the computer system (1) for which includes generating a 6D model of the skin surface (20) (S4), the method according to claim 3.

5. A computer system (1) including a probabilistic ranking of dermatological diagnostic results related to skin diseases, problems with the skin, and / or skin types in the dermatological evaluation result, the method including generating a dermatological evaluation result (S5), the method according to any one of claims 1 to 4.

6. Using a 6D model of the skin surface (20) as an input to a neural network trained to identify skin diseases, problems with the skin, and / or skin types, by using a plurality of 6D models of the skin surface from a plurality of people to generate a dermatological evaluation result (S5), the computer system (1) further including the 6D model of the skin surface including a plurality of surface data points with 3D coordinates and RGB color information, the method according to any one of claims 1 to 5.

7. Generating a 5D map of the skin surface (20) by applying a projection to the 6D model of the skin surface (20) (S52), the 5D map of the skin surface (20) including a plurality of map data points, each map data point including 2D coordinates and RGB color information; and using a plurality of 5D maps of the skin surface from a plurality of people as an input to a neural network trained to identify skin diseases, problems with the skin, and / or skin types to generate a dermatological evaluation using the 5D map of the skin surface (20) (S5), the 5D map of the skin surface including a plurality of map data points with 2D coordinates and RGB color information, the computer system (1) further including the method according to any one of claims 1 to 6.

8. Generating one or more 6D sub-models of the skin surface (20) by extracting one or more regions of a 6D model of the skin surface (20) (S51), wherein each of the 6D sub-models of the skin surface (20) comprises a plurality of surface data points of each region, and each surface data point comprises 3D coordinates and RGB color information; and generating a dermatological evaluation result using one or more 6D sub-models of the skin surface (20) as an input to a neural network trained to identify skin diseases, skin problems, and / or skin types using a plurality of 6D sub-models of the skin surface from a plurality of people (S5), wherein the one or more 6D sub-models of the skin surface comprise a plurality of surface data points of each region with 3D coordinates and RGB color information, the computer system (1) further comprising the method according to any one of claims 1 to 7.

9. Generating one or more 6D submodels of the skin surface (20) or at least one 5D map (20) of the skin surface (S51, S52), wherein one or more 6D submodels of the skin surface (20) are generated by extracting one or more regions of the 6D model of the skin surface (20), each of the 6D submodels of the skin surface (20) comprising a plurality of surface data points of the respective region, each surface data point comprising 3D coordinates and RGB color information, and the 5D map of the skin surface (20) being generated by applying a projection onto the 6D model of the skin surface (20), the 5D map of the skin surface (20) comprising a plurality of map data points, each map data point comprising 2D coordinates and RGB color information; and generating a dermatological evaluation result (S5) using at least one of a 6D model of the skin surface (20), one or more 6D submodels of the skin surface (20), or a 5D map of the skin surface (20) as an input to a neural network trained to identify skin diseases, skin problems, and / or skin types, the computer system (1) further comprising the method according to any one of claims 1 to 8.

10. Using a 6D model of the skin surface (20) to determine skin surface characteristics including at least one of wrinkles, wrinkle dimensions, pores, pore dimensions, milia, nodules, nodule dimensions, nodule shape, macula, macula dimensions, macula shape, papules, papule dimensions, papule shape, spots, spot dimensions, spot shape, pustules, pustule dimensions, blisters, blister dimensions, eruptions, eruption dimensions, eruption shape, acne, erosions, erosion dimensions, erosion shape, ulcers, ulcer dimensions, ulcer shape, crusts, scales, scale types, skin cracks, or atrophy; and generating a dermatological evaluation result (S5) using the skin surface characteristics, the computer system (1) further comprising the method according to any one of claims 1 to 9.

11. When the first dermatological evaluation of the first 6D model of the skin surface (20) is completed, storing the first 6D model of the skin surface (20) and the first dermatological evaluation result in the data storage; when receiving the second 6D model of the skin surface (20), identifying the first dermatological evaluation result in the data storage by comparing the first 6D model of the skin surface (20) with the second 6D model of the skin surface (20) (S58); when the second dermatological evaluation of the second 6D model of the skin surface (20) is completed, generating a tracking report (S60) by comparing the first dermatological evaluation result with the second dermatological evaluation result (S59); and further including a computer system (1) that executes drawing the tracking report on the user interface (103), the method according to any one of claims 1 to 10.

12. A computer system (1) that draws a 6D model of the skin surface (20) on a display device, receives evaluation data from a user via a user interface (103, 104), and generates a dermatological evaluation result using the evaluation data (S5), the method according to any one of claims 1 to 11, which includes generating a dermatological evaluation result (S5).

13. Generating the dermatological evaluation result includes an electronic device (10) of the computer system (1) that transmits the 6D model of the skin surface (20) to the processing system (11) of the computer system (1) via a communication network (2) (S7), the processing system (11) generates a dermatological evaluation result using the 6D model of the skin surface (20) received from the electronic device (10) (S5), and the processing system (11) transmits the dermatological evaluation result to the electronic device (10) (S8), the method according to any one of claims 1 to 12.

14. A computer system (1) including a camera (101), a depth sensor (102), user interfaces (103, 104), and one or more processors (100, 100*), the following: A step (S1) of controlling a camera (101) and a depth sensor (102) to record 3D surface images of a plurality of skin surfaces (20), wherein each 3D surface image is captured from different angles (α1 / β1, α2 / β2) and different distances (d1, d2) with respect to the skin surface (20) and includes a plurality of pixels with depth information and RGB color information; A step (S2) of detecting skin surface features (F1, F2, F3) in the 3D surface image; A step (S3) of determining the orientation and distance of each angle of the 3D surface image using the skin surface features (F1, F2, F3) and the depth information; A step (S4) of generating a 6D model of the skin surface (20) using the 3D surface images and their respective angles of orientation and distances, wherein the 6D model includes a plurality of surface data points, and each surface data point includes 3D coordinates and RGB color information; A step (S5) of generating a dermatological evaluation of the 6D model of the skin surface (20); A step (S6) of rendering the dermatological evaluation on a user interface (103); A computer system in which one or more processors (100, 100*) are configured to execute the steps above.

15. The computer system (1) according to claim 14, wherein one or more processors (100, 100*) are configured to execute the method according to any one of claims 1 to 12.

16. The computer system (1) according to claim 14 or 15, comprising an electronic device (10) and a processing system (11), wherein at least one of one or more processors (100) configured to transmit a 6D model (20) of the skin surface to the processing system (11) via a communication network (2) (S7) is included in the electronic device (10), and at least another one of one or more processors (100*) configured to generate a dermatological evaluation (S5) using the 6D model of the skin surface (20) received from the electronic device (10) and to transmit the dermatological evaluation to the electronic device (10) (S8) is included in the processing system (11).

17. Control the processor (100) of an electronic device (10) including a camera (101), a depth sensor (102), and a user interface (103) connected to the processor (100), and cause the processor (100) to perform the following: Control the camera (101) and the depth sensor (102) to record a plurality of 3D surface images of the skin surface (20) of human skin (step S1), wherein each 3D surface image is taken from different angles (α1 / β1, α2 / β2) and different distances (d1, d2) with respect to the skin surface (20) and includes a plurality of pixels with depth information and RGB color information; Detect skin surface features in the 3D surface images (step S2); Determine the orientation and distance of each angle of the 3D surface image using the skin surface features (F1, F2, F3) and the depth information (step S3); Generate a 6D model of the skin surface (20) using the 3D surface images and their respective angular orientations and distances (step S4), wherein the 6D model surface of the skin surface (20) includes a plurality of data points, and each surface data point includes 3D coordinates and RGB color information: Transmit the 6D model of the skin surface (20) to a computerized processing system (11) (step S7); Receive a dermatological evaluation result of the 6D model of the skin surface (20) from the computerized processing system (11) (step S8); Draw the dermatological evaluation result on the user interface (103) (step S6), A computer program product comprising computer program code configured to cause the above to be executed.

18. The computer program product according to claim 17, wherein the computer program code is further configured to control the processor (100) to execute the method according to any one of claims 1 to 12.

19. Control one or more processors (100*) of a computerized processing system (11) to perform the following: Receive a 6D model of a human skin surface (20) from an electronic device (10) (step S7), wherein the 6D model of the skin surface (20) includes a plurality of surface data points, and each surface data point includes 3D coordinates and RGB color information; Generate a dermatological evaluation result of the 6D model of the human skin surface (20) (step S5); A step of transmitting (S8) the dermatological evaluation result to the electronic device (10) A computer program product comprising computer program code configured to perform.

20. The computer program product according to claim 19, further configured with computer program code to control one or more processors (100*) to execute the method according to any one of claims 5 to 12.

Images