Calibration of Digital Images of Skin Tissue

Abstract

A computer-implemented method (100) for calibrating a digital image (110) of skin tissue. The digital image includes a calibration marker (120) located on or near the skin tissue, and the calibration marker includes benchmark elements (121-131) characterized by at least one benchmark attribute value of at least one attribute type. The computer-implemented method includes a step (101) of detecting the calibration marker (220) in the digital image (110), and a step (102) of detecting the benchmark elements (121-131) in a cropped portion (111) of the digital image including the calibration marker. The computer-implemented method further includes, for each benchmark element (121-131), identifying at least one depicted attribute value (113) of at least one attribute type based on the pixels of the digital image located within each benchmark element (step 103), and calibrating the digital image by correcting the deviation between the benchmark attribute value (114) associated with each benchmark element and the depicted attribute value (113) (step 104).

Description

Technical Field

[0001] The present invention generally relates to the calibration of digital images, particularly to the calibration of digital images of skin tissue.

Background Art

[0002] In some skin diseases, such as psoriasis and dermatitis, it is necessary to evaluate skin lesions in order to determine the severity and progression of the skin disease. This evaluation is usually performed manually by experts over time, but there is a high likelihood of variation among experts. Therefore, software applications have been developed to at least partially automate this evaluation based on digital images of skin lesions taken by patients using smartphones or the like. In that method, visible signs indicating the severity and progression of skin lesions, such as the color and size of the lesion, may be significantly distorted within the digital image, and in that case, there is a problem that the skin disease cannot be optimally evaluated.

Summary of the Invention

[0003] An object of the present invention is to solve or mitigate the above-described problems and issues, particularly by calibrating digital images of skin tissue.

[0004] According to a first aspect, this object is achieved by a computer-implemented method for calibrating a digital image of skin tissue, the digital image including a calibration marker located on or near the skin tissue. The calibration marker includes a benchmark element characterized by at least one benchmark attribute value of at least one attribute type. The computer-implemented method comprises - detecting a calibration marker within the digital image; - detecting a benchmark element within a cropped portion of the digital image that includes the calibration marker; - for each benchmark element, identifying at least one depicted attribute value of at least one attribute type based on the pixels of the digital image located within each benchmark element; calibrating the digital image by correcting the deviation between the benchmark attribute values associated with each benchmark element and the depicted attribute values.

[0005] The calibration marker can be included in the digital image by placing the calibration marker on or near the skin tissue when photographing the digital image of the skin tissue. The calibration marker may be a piece of paper or a plastic film provided with one or more benchmark elements. The calibration marker may be, for example, a transparent plastic film printed with a plurality of benchmark elements.

[0006] By detecting the calibration marker in the digital image, a cropped portion of the digital image including the calibration marker can be obtained. In other words, the portion of the digital image including the calibration marker can be identified. The cropped portion enables the detection of the benchmark elements to be performed more quickly and efficiently. This is because the portion of the digital image excluding the cropped portion is omitted from the detection of the benchmark elements. In other words, the search space for detecting one or more benchmark elements is significantly reduced.

[0007] Each benchmark element is characterized by one or more benchmark attribute values of one or more attribute types. In other words, one benchmark element can be characterized by a plurality of benchmark attribute values associated with each attribute type. The attribute type can be, for example, color, position, dimension, surface area, pixel density, orientation or shape. The benchmark attribute value refers to a known standard or reference point of a specific attribute type. The described attribute value refers to the value of a specific attribute type represented or rendered in a digital image. By comparing the described attribute value with the benchmark attribute value, it becomes possible to obtain the deviation between the appearance of the skin tissue represented or rendered in the digital image and the appearance of the actual skin tissue. By correcting these deviations, a calibrated or normalized digital image is obtained in which the skin tissue is rendered almost accurately, that is, represented in the digital image in the same way as an observer recognizes the actual skin tissue. Therefore, one or more benchmark elements of the calibration marker function as a reference for calibrating the digital image.

[0008] By calibrating the digital image, a calibrated representation of visible signs such as the color and size of the lesion, which indicate the severity and progression of the skin disease, can be obtained. This has the advantage that the skin disease can be evaluated more accurately, reliably and objectively based on the digital image of the skin tissue. The calibrated digital image of the skin tissue can be used for training a machine learning model or, alternatively, has the further advantage that it can be used as an input to a machine learning model for determining the severity and / or progression of the skin disease.

[0009] According to one embodiment, the detection of the calibration marker can be performed by a machine learning model trained to detect the calibration marker in the digital image.

[0010] The detection of calibration markers can be achieved, for example, by object detection or object recognition. The machine learning model can be obtained by training a classifier based on digital images of annotated skin tissue containing calibration markers. Here, the calibration markers may be labeled. The machine learning model may be based on, for example, a neural network, a support vector machine, or a convolutional neural network. Alternatively, the machine learning model can be obtained by unsupervised learning or reinforcement learning.

[0011] According to one embodiment, the detection of benchmark elements can be performed by a machine learning model trained to assign each pixel within a cropped portion of a digital image to a benchmark element.

[0012] For this purpose, the machine learning model can be configured to classify individual pixels within a cropped portion of a digital image into respective classes associated with benchmark elements, for example, by semantic image segmentation. By doing so, it becomes possible to generate a pixel map or mask that can identify the pixels that are part of each benchmark element, thereby enabling the detection of benchmark elements.

[0013] According to one embodiment, one or more benchmark elements can be characterized by respective benchmark color values of color types, and the step of calibrating the digital image includes the step of correcting the color values of the pixels within the digital image.

[0014] The color values of the color type can, in particular, indicate hue, lightness, saturation, and / or luminance. The color values can be values according to any color model such as, for example, the RGB color model, the CMYK color model, the YUV color model, the HSL color model, the HSV color model, etc. Therefore, one or more benchmark elements within the calibration marker can be characterized by preset color values, i.e., benchmark color values. Preferably, the calibration marker includes a plurality of benchmark elements each characterized by a respective benchmark color value. Preferably, each benchmark color value includes skin tones such as, for example, [0,20,10,0] CMYK, [0,20,30,0] CMYK, [0,30,30,50] CMYK, and [0,40,40,50] CMYK.

[0015] Thereby, the color values of the benchmark elements depicted in the digital image can be compared with the benchmark color values to obtain a deviation in the color values. This color deviation can be significantly corrected for calibrating the digital image. Correcting the color values can include adjusting the color values of the pixels in the digital image such that each color deviation associated with the various benchmark color values is minimized. In other words, the color deviation associated with a benchmark color value is minimized to such an extent that the color deviations associated with other benchmark color values do not significantly deteriorate or increase due to the adjustment. For this reason, the correction of the color values of the digital image is a multivariate optimization aimed at minimizing the color deviations associated with each benchmark element. Thereby, when correcting the color deviation of a specific color value such as, for example, red, it is possible to avoid additional deviations or color shifts in other color values such as, for example, blue or green. This can be achieved, for example, by an optimization method based on partial least squares regression.

[0016] By calibrating the color values, it becomes possible to more reliably evaluate skin tissue based on a digital image, regardless of factors affecting the color of the digital image such as ambient lighting and chromatic aberration. This has the advantage that the color of skin lesions can be reproduced almost accurately within the digital image, regardless of the conditions under which the digital image was taken or the device used to take the digital image. Furthermore, there is also the advantage that the severity and / or progression of skin diseases can be determined more reliably based on the digital image.

[0017] According to one embodiment, the step of identifying at least one depicted attribute value of a color type includes the step of obtaining an average depicted color value of pixels of a digital image located within each of one or more benchmark elements.

[0018] The average depicted color value can be obtained by identifying the depicted color values of each pixel within each benchmark element and obtaining the average of those depicted color values.

[0019] According to one embodiment, a computer-implemented method can further include the step of obtaining each color deviation between the average depicted color value and a benchmark color value associated with each of one or more benchmark elements.

[0020] According to one embodiment, at least four benchmark elements can be characterized by benchmark position values of a position type, and the step of calibrating the digital image can include the step of adjusting the viewpoint of the digital image.

[0021] At least four benchmark elements within the calibration marker can have preset positions on the calibration marker. Each benchmark position value may indicate those preset positions. Each benchmark position value may be coordinates according to a coordinate system having, for example, a first axis along a vertical edge of the calibration marker and a second axis along a horizontal edge of the calibration marker.

[0022] By adjusting the perspective of a digital image, it becomes possible to more reliably evaluate skin tissue based on the digital image, regardless of factors affecting the perspective of the digital image such as optical aberrations or camera angles. This has the advantage that the size and shape of skin lesions can be reproduced almost exactly within the digital image, regardless of the conditions under which the digital image was taken or the equipment used to take the digital image. Furthermore, since the shape and size of skin lesions are typical visible signs of skin diseases, there is also the advantage that the severity and / or progression of skin diseases can be determined more reliably.

[0023] According to one embodiment, the step of identifying at least one depicted attribute value of a position type includes the step of identifying the depicted positions of at least 4 benchmark elements each.

[0024] Any pixel or point within each of at least 4 benchmark elements characterized by a benchmark position value can indicate the depicted position of the benchmark element. For example, the pixel at the center of each benchmark element can indicate the position of the benchmark element. Therefore, identifying the depicted position includes identifying the position of the pixel.

[0025] According to one embodiment, the computer-implemented method further includes the step of determining a depicted polygon defined by the depicted positions of at least 4 benchmark elements each within the digital image, and the step of adjusting the perspective of the digital image can further include the step of mapping the pixels within the depicted polygon to the pixels within a benchmark polygon defined by the benchmark position values of at least 4 benchmark elements.

[0026] Therefore, at least four benchmark elements characterized by benchmark position values can define a benchmark polygon. In other words, each corner of the benchmark polygon can be defined by the respective benchmark position values of at least four benchmark elements. Similarly, each corner of the depicted polygon can be defined by the respective depicted positions of at least four benchmark elements. The viewpoint of the digital image can be adjusted based on the benchmark polygon and the depicted polygon. This can be achieved by a geometric image transformation that distorts the depicted polygon into the shape of the benchmark polygon. That is, the pixel grid of the digital image is deformed based on the deviation between the depicted polygon and the benchmark polygon.

[0027] According to one embodiment, one or more benchmark elements can define a reference axis of a calibration marker, and the step of calibrating the digital image includes the step of rotating the digital image based on the reference axis.

[0028] The calibration marker includes a plurality of benchmark elements arranged in a row, thereby being able to define a reference axis. The reference axis can be defined, for example, by each preset pixel or point within the plurality of benchmark elements arranged in a row, such as the geometric center of each benchmark element. The plurality of benchmark elements can be further characterized by one or more additional attribute values of various attribute types, such as color values. This has the further advantage that the digital image can be rotated based on the reference axis without providing additional benchmark elements in the surface area of the calibration marker. Alternatively, the calibration marker can include a benchmark element substantially shaped as a line or arrow indicating the reference axis.

[0029] According to one embodiment, the computer-implemented method can further include the step of determining an angle between a reference axis of a calibration marker and a preset edge of a digital image.

[0030] The preset edge 513 can be any outer edge of the digital image 510. Thereby, by rotating the digital image so that the reference axis is oriented substantially horizontally or vertically, the digital image can be rotated to a calibrated position. For this purpose, the digital image can be rotated at an angle corresponding to the determined angle between the reference axis and the preset edge.

[0031] According to one embodiment, the reference axis can be characterized by the benchmark position on the calibration marker, and the rotation of the digital image can be further performed based on the depicted position of the reference axis.

[0032] The reference axis can be arranged on the surface of the calibration marker so that the calibration marker is asymmetric. For example, the reference axis can be arranged on the calibration marker offset from the outer boundary of the calibration marker. Thereby, the calibration marker can be oriented in a preset direction. Thereby, it is possible to avoid the digital image being rotated in an inverted orientation.

[0033] According to one embodiment, one or more benchmark elements can be characterized by the surface area type benchmark surface area value, and the computer-implemented method can further include the step of determining the surface area associated with each pixel of the digital image based on the number of pixels located within each of the one or more benchmark elements and the benchmark surface area of each of the one or more benchmark elements.

[0034] In other words, one or more benchmark elements can be characterized by a preset surface area or a known surface area, i.e., a benchmark surface area value. Thus, for each pixel within those one or more benchmark elements in the digital image, the depicted surface area can be determined. This can be achieved, for example, by dividing the benchmark surface area value of each benchmark element by the number of pixels within each benchmark element. Thereby, based on the number of pixels of the skin lesion depicted in the digital image, it becomes possible to determine the size or dimension of the skin lesion. This has the further advantage that, since the size of the skin lesion is a typical visible sign of a skin disease, the severity and / or progression of the skin disease can be more reliably determined based on the digital image.

[0035] According to one embodiment, determining the surface area associated with pixels located outside one or more benchmark elements can include interpolating and / or extrapolating the surface area associated with pixels located within the one or more benchmark elements.

[0036] In other words, for pixels not located within one or more benchmark elements characterized by the benchmark surface area, the surface area per pixel can be determined. For pixels in the digital image located between at least the first and second benchmark elements, this can be achieved by interpolating the surface area associated with the pixels located within each benchmark element. Thereby, the size or dimension of the skin lesion located between the benchmark elements (e.g., inside the boundary of the calibration marker) can be accurately determined. For pixels in the digital image that are not located between at least two benchmark elements, for example, pixels at a considerable distance from the calibration marker, this can be achieved by extrapolating the surface area associated with the pixels located within each benchmark element. Thereby, the size or dimension of the skin lesion not located between the benchmark elements, for example, the skin lesion outside the boundary of the calibration marker, can be accurately determined.

[0037] According to a second aspect, the present invention relates to a data processing system configured to execute the computer-implemented method according to the first aspect.

[0038] According to a third aspect, the present invention relates to a computer program comprising instructions which, when executed by a computer, cause the computer to execute the computer-implemented method according to the first aspect.

[0039] According to a fourth aspect, the present invention relates to a computer-readable medium comprising instructions which, when executed by a computer, cause the computer to execute the computer-implemented method according to the first aspect.

Brief Description of the Drawings

[0040]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

[0041] Skin diseases such as psoriasis and dermatitis are usually evaluated using a scoring system or a scoring tool. Those scoring systems typically combine an assessment of the severity of skin lesions and the extent of the skin area affected by the skin lesions into one score. A skin lesion refers to an area of skin tissue or a part of skin tissue that has characteristics significantly different from the surrounding skin tissue (e.g., different color, shape, size, or texture). Examples of scoring systems include the Psoriasis Area and Severity Index (PASI), the Scoring Atopic Dermatitis (SCORAD), etc. By repeatedly evaluating skin diseases regularly using such scoring systems, it is usually possible to identify the progression of skin diseases over time, thereby making it possible to determine the treatment effect.

[0042] Typically, a scoring system is based on the interpretation of visible signs (also called visible indicators or clinical signs) that indicate the extent and severity of a skin disease. Such visible signs typically include erythema or redness, sclerosis or thickening, desquamation or scaling, swelling, the effects of scratching, exudation, crust formation, lichenification, dryness, etc. Furthermore, the extent of the affected area contributes to the assessment of the skin disease. That is, the size or dimension of the skin lesion indicates the severity of the skin disease.

[0043] The problem with such a scoring system is that, because the evaluation is subjective, variation between observers is likely to occur and the results may differ depending on the person performing the evaluation. Furthermore, the evaluation of skin diseases based on a scoring system is usually performed by trained experts and takes time. Therefore, software applications have been developed that at least partially automate this evaluation based on digital images of skin tissue containing skin lesions. Those digital images are preferably taken by the patient himself / herself, for example using a smartphone or tablet. In this case, there is a problem that visible signs indicating the severity of the skin disease are significantly distorted in the digital image, resulting in an inaccurate evaluation of the skin disease. For example, the color of the digital image may be distorted due to ambient lighting and / or chromatic aberration. The size, dimension, focus and / or viewpoint of the digital image may be distorted, for example due to monochromatic aberration, by taking the digital image at an inappropriate camera angle. Therefore, it may be desirable to calibrate or normalize the digital image of the skin tissue.

[0044] FIG. 1 shows step 100 of a computer-implemented method according to an embodiment for calibrating a digital image 110 of skin tissue. The digital image 110 can be obtained, among other things, by a camera included in a smartphone, a tablet, a web camera, or a digital single-lens reflex camera. The digital image 110 includes a calibration marker 120 located on or near the skin tissue. The calibration marker 120 can be included in the digital image 110 by placing the calibration marker on or near the skin tissue when taking the digital image 110. For example, a person with a skin disorder or skin disease can place the calibration marker 120 on or near the skin tissue, for example, near a skin lesion, and use the camera of their smartphone to obtain the digital image 110.

[0045] The calibration marker 120 may be a very thin strip or film including one or more benchmark elements 121-131. The calibration marker 120 may be made of, for example, paper or plastic. The one or more benchmark elements may be printed on the calibration marker 120. The calibration marker 120 further includes an unmarked body 132, that is, a portion of the calibration marker 120 where the benchmark elements 121-131 are substantially absent. The calibration marker 120 may be made of a transparent material so that the skin tissue below the unmarked body 132 of the calibration marker 120 can still be seen even when covered by the calibration marker. The calibration marker 120 may be made of, for example, a transparent plastic film, laminated polyethylene transparent paper, and / or polypropylene. Alternatively or complementarily, a part of the unmarked body 132 may be made to substantially contain no material so that the skin tissue within this part can be visually recognized when covered by the calibration marker.

[0046] Each of the benchmark elements 121 - 131 is characterized by one or more benchmark attribute values of one or more attribute types. In other words, one benchmark element 121 - 131 can be characterized by a plurality of benchmark attribute values of various attribute types. The attribute type can be, for example, color, position, dimension, surface area, pixel density, orientation or shape. Thus, one benchmark element 121 - 131 can be characterized by, for example, a color value, a position value and a surface area. The benchmark attribute value 114 refers to a known standard or reference point of a specific attribute type, that is, a fixed preset value. Therefore, each of the benchmark elements 121 - 131 can be characterized by, for example, a known color value, a known position, a known dimension or a known surface area, etc.

[0047] In the first step 101, the calibration marker 120 is detected within the digital image 110. In so doing, a cropped portion 111 of the digital image 110 that includes the calibration marker 120 is obtained. The cropped portion 111 simply refers to an identified sub - section or a part within the digital image 110 that includes at least the calibration marker 120, and it will be apparent that it can include removing other parts of the digital image 110 excluding the calibration marker 120, but does not necessarily have to.

[0048] Detecting the calibration marker 120 from the digital image 110 in step 101 may be performed by a machine learning model trained to detect the calibration marker 120 within the digital image. This can be achieved, for example, by object detection or object recognition. The trained machine learning model may be obtained by supervised learning in which a training dataset is provided to the classifier. This training dataset can include a substantial amount of annotated digital images containing calibration markers. Thus, the positions of the calibration markers within each of the annotated digital images of the training dataset may be labeled or marked. The machine learning model may be based on, for example, a neural network, a support vector machine, or a convolutional neural network. Alternatively or complementarily, the trained machine learning model may be obtained by unsupervised learning or reinforcement learning.

[0049] In the next step 102, one or more benchmark elements 121-131 are detected within the cropped portion 111 of the digital image. Since the portion of the digital image 110 excluding the cropped portion is omitted from the detection of the benchmark elements, the cropped portion enables the benchmark elements 121-131 to be detected more quickly and efficiently. That is, the search space for detecting the benchmark elements 121-131 is significantly reduced from the entire digital image 110 to the cropped portion 111.

[0050] The detection of these benchmark elements 121-131 can be performed by a machine learning model trained to assign each pixel within the cropped portion 111 of the digital image to each of the benchmark elements 121-131. Thus, the machine learning model can be configured or trained to classify individual pixels within the cropped portion 111 of the digital image 110 into respective classes associated with each of the benchmark elements 121-131. Further, the machine learning model can be configured or trained to classify individual pixels within the cropped portion 111 into classes associated with the body 132 of the calibration marker 120 without markings. This can be achieved, for example, by semantic image segmentation. Here, a pixel map or mask is generated that assigns or classifies each pixel within the digital image of the calibration marker 120 to a respective class. The pixel map or mask can be generated by manually labeling individual pixels within the digital image of the calibration marker 120. Thereby, individual pixels within the digital image can be associated with various benchmark elements 121-131, thereby detecting the benchmark elements.

[0051] In the next step 103, based on the pixels of the digital image 110 located within each of the benchmark elements 121-131, at least one described attribute value of at least one attribute type is determined for each of the benchmark elements 121-131. The described attribute value 113 refers to the value of a specific attribute type represented or rendered within the digital image 110. As described above, the attribute value can be greatly distorted within the digital image, for example, the color can be shifted or the perspective can be distorted. Therefore, by comparing the described attribute value 114 with the known benchmark attribute value 113, the degree of such color shift and distortion can be determined. Thus, the deviation between the appearance of the skin tissue represented or rendered within the digital image 110 and the appearance of the actual skin tissue can be determined.

[0052] In the next step 104, the digital image 110 is calibrated by correcting the deviation between the depicted attribute value 114 and the benchmark attribute value 113 associated with each of the benchmark elements 121 - 131. It will be clear that only the depicted attribute value 114 and the benchmark attribute value 113 of the same attribute type can be compared to obtain the deviation. By correcting these deviations, a calibrated or normalized digital image 115 is obtained, and the skin tissue is rendered in a substantially true-to-life form. That is, the skin tissue is represented in the digital image in the same way that an observer recognizes the actual skin tissue. In other words, by substantially correcting the deviation from reality, the representation of the skin tissue in the digital image is made objective. In this way, one or more of the benchmark elements 121 - 131 of the calibration marker 120 function as a reference for calibrating the digital image 110.

[0053] By calibrating the digital image 110, a calibrated representation of the visible signs indicating the severity and progression of skin diseases, such as the color and size of lesions, can be obtained. This has the advantage that the skin disease can be evaluated more accurately, reliably, and objectively based on the digital image of the skin tissue. The calibrated digital image of the skin tissue can be used for training a machine learning model or, alternatively, as an input to a machine learning model for determining the severity and / or progression of skin diseases. Additionally, there is the advantage that the patient himself / herself can efficiently evaluate the skin disease.

[0054] Figure 2 shows step 200 of a computer-implemented method according to an embodiment where calibration of a digital image includes correction of color values within the digital image. One or more benchmark elements 121, 122, 123 within the calibration marker can be characterized by respective benchmark color values 221 of a color type. Those benchmark elements 121, 122, 123 can be arranged on the calibration marker so as to form a circular or ring-shaped arrangement of the benchmark elements 121, 122, 123, i.e., a color roulette 210 or color wheel. The inner region 211 surrounded by the ring-shaped arrangement of the benchmark elements 121, 122, 123 is substantially free of benchmark elements. That is, the inner region 211 forms part of the unmarked body of the calibration marker. The inner region 211 may be substantially free of material. In other words, the calibration marker can include a notch or through-hole having substantially the same shape as the inner region 211. Alternatively, the benchmark elements 121, 122, 123 can also be provided in any other shape that provides an inner region substantially free of benchmark elements, such as a rectangular arrangement, an ellipsoidal arrangement, or a stadium arrangement.

[0055] The color values of the benchmark elements 121, 122, 123 can indicate, in particular, hue, lightness, saturation and / or luminance. The color values can be values according to any color model, such as, for example, the RGB color model, the CMYK color model, the YUV color model, the HSL color model, the HSV color model, etc. Thus, the benchmark elements 121, 122, 123 are characterized by preset or known color values, i.e., the benchmark color values 221.

[0056] Each benchmark color value 221 of the benchmark elements 121, 122, 123 can include a skin tone or a skin-like color that matches various skin phototypes. For example, each benchmark color value 221 may correspond to a Fitzpatrick scale level. Thereby, a digital image can be calibrated more accurately for skin tissue. Each benchmark color value 221 can further include a non-skin color such as a pure pigment value, for example. The calibration markers can include, for example, 12 benchmark elements 121, 122, 123 characterized by their respective benchmark color values according to the CMYK color model, namely, [0,20,10,0], [0,30,20,0], [0,30,30,0], [0,20,30,0], [0,20,30,10], [0,30,20,20], [0,20,30,20], [0,20,30,30], [0,30,30,50], [0,20,30,50], [0,40,40,50], and [0,40,40,70].

[0057] Step 200 can be executed after detecting the benchmark elements in step 102 of FIG. 1. In the first step 201, for each benchmark element 121, 122, 123 characterized by the benchmark color value 221, a descriptive color value is determined. Determining the descriptive color value of the benchmark element 122, i.e., the descriptive attribute value of the color type, may include determining the average descriptive color value 220 of the benchmark element 122. This can be obtained by identifying the color values of each pixel 214 within the benchmark element 122 and determining the average color value 220 of all the pixels 213 included in the benchmark element 122. The average can be, in particular, any of an arithmetic mean, a median, or a mode.

[0058] In the next step 202, the average description color value 220 of each benchmark element 121, 122, 123 characterized by the benchmark color value 221 can be compared with the benchmark color value 221 associated with each respective benchmark element. For example, the average description color value 220 of the benchmark element 121 can be compared with the benchmark color value 221 associated with that benchmark element 121. Thereby, the color deviation 222 of each benchmark element 121, 122, 123 characterized by the benchmark color value 221 can be obtained. In other words, the deviation of the digital image from each benchmark color value 221 can be obtained.

[0059] In step 203, those color deviations 222 can be substantially corrected to calibrate the color of the digital image. The correction of the color value includes adjusting the color values of the pixels in the digital image such that each color deviation 222 associated with the various benchmark color values 221 is minimized. In other words, the color deviation 222 associated with the benchmark color value 221 is minimized to such an extent that the color deviation 222 associated with another benchmark color value 221 does not significantly deteriorate or increase due to the adjustment. For this reason, the correction of the color value of the digital image is a multivariate optimization aimed at minimizing the color deviation 222 associated with each benchmark element 121, 122, 123. Thereby, when correcting the color deviation of a specific color value, such as red, it is possible to avoid further deviation or color shift in other color values, such as blue or green. The determination of the color correction for calibrating the digital image can be realized, for example, by an optimization method based on partial least squares regression. Alternatively, the correction of the color value may include adjusting the color values of the pixels in the digital image such that the color deviation becomes substantially zero.

[0060] Therefore, through color correction in step 203, a color-corrected digital image 223 is obtained. Thereby, it becomes possible to more reliably evaluate skin tissue based on the digital image, regardless of factors affecting the color of the digital image such as ambient lighting and chromatic aberration. This has the advantage that the color of the skin lesion part is reproduced almost accurately within the color-corrected digital image 223, regardless of the conditions under which the digital image was taken or the device used to take the digital image. Furthermore, there is also the advantage that the severity and / or progression of skin diseases can be determined more accurately based on the digital image.

[0061] FIG. 3 shows step 300 of a computer-implemented method according to an embodiment, where the calibration of the digital image includes adjustment of the perspective of the digital image. At least four benchmark elements 124, 125, 126, 127 included in the calibration marker 320 may be characterized by position-type benchmark position values 314. Thereby, those benchmark elements 124, 125, 126, 127 can have preset positions on the surface of the calibration marker 320 indicated by their respective benchmark position values 314. The benchmark position values 314 may be coordinates according to a coordinate system having, for example, a first axis 331 along the vertical edge of the calibration marker 320 and a second axis 332 along the horizontal edge of the calibration marker 320.

[0062] The benchmark elements 124, 125, 126, 127 characterized by the benchmark position values 314 can be arranged near the outer boundary of the calibration marker, for example, at positions substantially close to the corners or edges of the calibration marker 320. The benchmark elements 124, 125, 126, 127 can include an outer band 310 that defines at least a part of the boundary of the inner band 311. The inner band 311 can further surround an inner area 312. The outer band 310, the inner band 311, and the inner area 312 can further have substantially contrasting colors, such as black and white. Thereby, the detection of the benchmark elements 124, 125, 126, 127 can be improved.

[0063] The benchmark position values 314 of the respective benchmark elements 124, 125, 126, 127 can be defined by any point or pixel within each benchmark element, for example, the pixel located at the geometric center of the benchmark element, the pixel located at the geometric center of the inner area 312, or the pixel located at the corner of the inner band 311. The benchmark position values 314 associated with the respective benchmark elements 124, 125, 126, 127 can also define a benchmark polygon 321. For example, the benchmark position values 314 can define the corners of a rectangle. In other words, each benchmark position value 314 can define the boundary of the benchmark polygon 321. The benchmark polygon 321 may be another polygon according to the number and arrangement of the benchmark elements 124, 125, 126, 127, for example, a hexagon defined by 6 benchmark elements. Alternatively or complementarily, the benchmark polygon 321 can include additional benchmark elements (not shown in FIG. 3) that do not define the corners of the polygon. Such additional benchmark elements may be arranged, for example, on the sides of the benchmark polygon 321 or within the area enclosed by the polygon 321. For example, the benchmark polygon 321 can include two additional benchmark elements respectively arranged on the sides between elements 124 and 126 and on the sides between elements 125 and 127.

[0064] Step 300 can be executed after detecting benchmark elements 124, 125, 126, 127 in step 102 of FIG. 1. In the first step 301, the depicted position values 313 of elements 124, 125, 126, 127 can be determined as drawn in the digital image (340). In the next step 302, a drawn polygon 341 can be identified based on the depicted position values 313 of each benchmark element 124, 125, 126, 127. In other words, the benchmark elements 124, 125, 126, 127 drawn in the digital image can define the drawn polygon 341. For example, FIG. 3 shows a calibration marker 340 drawn in a digital image, but the calibration marker 320 is substantially distorted or deformed. This can occur, for example, when the digital image is not taken substantially perpendicular or substantially vertical to the skin tissue, thereby distorting the perspective between the skin tissue and the calibration marker in the digital image.

[0065] In the next step 303, based on the benchmark polygon 321 and the drawn polygon 341, the perspective of the digital image can be adjusted. This can be achieved by a geometric image transformation that distorts the drawn polygon 341 into the shape of the benchmark polygon 321. In other words, the pixel grid of the digital image is deformed based on the deviation between the drawn polygon 341 and the benchmark polygon 321. Thereby, the pixels within the drawn polygon 341 can be mapped to the pixels within the benchmark polygon 321 according to a geometric transformation matrix. By doing so, the geometric distortion of the digital image can be significantly corrected, thereby obtaining a perspective-calibrated image 315.

[0066] By adjusting the perspective of a digital image, it is possible to more reliably evaluate skin tissue based on the digital image regardless of factors affecting the perspective of the digital image such as optical aberration and camera angle. Further, by correcting the perspective of the digital image, it becomes possible to more accurately determine the size of a skin lesion. This has the advantage that, regardless of the conditions under which the digital image was taken or the device used to take the digital image, i.e., regardless of how the digital image was taken, the size and shape of the skin lesion can be reproduced almost accurately within the perspective-corrected digital image 315. This further has the advantage that, since the shape and size of the skin lesion are typical visible signs of skin diseases, the severity and / or progression of skin diseases can be determined more reliably.

[0067] FIG. 4 shows step 400 of a computer-implemented method according to an embodiment, which further includes steps of determining the surface area associated with each pixel of a digital image. For this purpose, one or more benchmark elements can be characterized by a surface area type of benchmark surface area value. The one or more benchmark elements characterized by the surface area value can include the benchmark elements 124, 125, 126, 127 of FIGS. 1 and 3 characterized by a position value. Alternatively or complementarily, the one or more benchmark elements characterized by the surface area value can include the benchmark elements 121, 122, 123 of FIGS. 1 and 2 characterized by a color value.

[0068] Step 400 can be executed after detecting each benchmark element in the digital image in step 102 of FIG. 1. In the first step 401, the surface area associated with each pixel within the benchmark element can be determined based on the number 410 of pixels located within the benchmark element and the benchmark surface area 411 of those benchmark elements. First, the number 410 of pixels within each benchmark element is determined. Thereafter, by dividing the benchmark surface area by the obtained number 410 of pixels, the surface area value of each pixel within the benchmark element, i.e., the pixel surface area 412, can be obtained.

[0069] It will be apparent that the pixel surface area 412 can be different for each benchmark element, for example, depending on the perspective of the digital image. For this reason, the surface area associated with the pixels located outside the benchmark element can also vary significantly depending on the perspective of the image. The pixels located outside the benchmark element can include pixels located within the body without the marking of the calibration marker or pixels not located on the calibration marker. The surface plot 420 shows an example of the surface area value 421 associated with each pixel in the digital image of the skin tissue. The axis 433 can indicate the magnitude of the surface area value, and the axes 431, 432 can respectively indicate the x - coordinate and y - coordinate of the pixel grid of the digital image. It is clear that a digital image of the skin tissue in which the pixels have surface area values following the surface plot 420 cannot be used to accurately determine the size or dimensions of the skin lesion without determining the surface area value 421 of each pixel in the digital image.

[0070] For this purpose, the computer-implemented method can further include, at step 402, interpolating the surface area associated with the pixels located within the benchmark elements. For example, points 423, 424, 425, 426 can represent respective benchmark elements characterized by respective pixel surface areas 412. By interpolating between the respective benchmark elements 423, 424, 425, 426, a more accurate surface area value for the pixels located within the polygon 422 defined by the elements 423, 424, 425, 426, such as the pixels located within the calibration marker, can be obtained. The elements 423, 424, 425, 426 can correspond, for example, to the benchmark elements 124, 125, 126, 127 in FIGS. 1 and 3. The polygon 422 can correspond, for example, to the polygon 341 depicted in FIG. 3. Thereby, the size or dimension of a skin lesion located between the benchmark elements, for example, inside the boundary of the calibration marker, can be accurately determined.

[0071] Alternatively or complementarily, the computer-implemented method can further include the step of extrapolating the surface area associated with the pixels located within the benchmark elements. This extrapolation may be further performed based on the interpolated surface area associated with the pixels located within the polygon 422. By extrapolating the surface area of the pixels, a more accurate surface area value can be obtained for the pixels located outside the polygon 422, such as the pixels located outside the calibration marker. Thereby, the size or dimension of a skin lesion not located between the benchmark elements, for example, a skin lesion located outside the boundary of the calibration marker, can be accurately determined.

[0072] It will be apparent that additional benchmark elements arranged within the polygon 422, such as those arranged between elements 423, 426 and between elements 424, 425, can further improve the accuracy of interpolation and / or extrapolation. Furthermore, it will be apparent that the interpolation in step 402 and the extrapolation in step 403 can be performed in any order or substantially simultaneously.

[0073] Figure 5 shows step 500 of a computer-implemented method according to an embodiment for rotating a digital image based on a reference axis 511. For this purpose, one or more benchmark elements define the reference axis 511 of the calibration marker 120. The calibration marker 120 includes a plurality of benchmark elements 130, 131 arranged in a row, whereby the reference axis 511 can be defined. The reference axis may be defined, for example, by each preset pixel or point within a plurality of benchmark elements arranged in a row. For example, FIG. 5 shows seven benchmark elements 130, 131 arranged such that the geometric centers of each benchmark element 130, 131 define the reference axis 511.

[0074] The plurality of benchmark elements 130, 131 may further be characterized by one or more additional attribute values of various attribute types, such as color values. For example, the seven benchmark elements 130, 131 in FIG. 5 can be characterized by their respective grayscale color values. Alternatively, the calibration marker 120 can include a benchmark element substantially shaped as a line or arrow indicating the reference axis 511.

[0075] Step 500 can be performed after detecting benchmark elements in the digital image in step 102 of FIG. 1. In the first step 502, a reference axis 511 can be determined from the detected benchmark element 501. In the next step 503, an angle 512 between the reference axis 512 and a preset edge 513 of the digital image 510 can be obtained. The preset edge 513 can be any outer edge of the digital image 510. In the next step 505, the digital image can be rotated to a calibrated position based on the angle 512. The digital image 510 can be rotated so that the reference axis faces substantially horizontally. For example, in FIG. 5, the digital image 510 can be rotated counterclockwise by an angle corresponding to the angle 512. Alternatively, for example, when the reference axis 511 is located in the longitudinal direction of the calibration marker 120 (i.e., the direction perpendicular to the reference axis 511 in FIG. 5), the digital image can also be rotated so that the reference axis faces substantially vertically.

[0076] The reference axis 511, i.e., the benchmark elements 130, 131, can be further characterized by the benchmark positions on the calibration marker 120. The reference axis 511 may be arranged on the surface of the calibration marker so that the calibration marker 120 is asymmetric. For example, the reference axis 511 is substantially arranged toward the outer boundary of the lower part of the calibration marker 120. Thereby, for example, the calibration marker can be oriented in a preset direction so that the reference axis is arranged at the lower part of the calibration marker 120. Thereby, it is possible to avoid the digital image 510 being rotated in an inverted orientation. For this purpose, in step 504, the depicted position of the reference axis is specified, and in step 505, in addition to the angle 512, the digital image can be rotated based on this depicted position compared with the benchmark position of the reference axis.

[0077] The computer-implemented method can further include the step of estimating the resolution of a digital image. For this purpose, the calibration marker can include a Siemens star 129. If the resolution does not meet the minimum threshold, the user can be prompted to repeat the capture of the digital image or to provide a digital image with a higher resolution. The estimation of the resolution of the digital image can be performed at any point in the computer-implemented method, for example, before detecting the calibration marker in step 101 of FIG. 1 or after calibrating the digital image in step 104 of FIG. 1. This can ensure that the resolution of the calibrated image 115 is sufficient for the evaluation of skin diseases based on the digital image.

[0078] Figure 6 shows a suitable computing system 600 that enables implementation of the above-described method embodiments according to the present invention. The computing system 600 can typically be formed as a suitable general-purpose computer and can include a bus 610, a processor 602, a local memory 604, one or more optional input interfaces 614, one or more optional output interfaces 616, a communication interface 612, a storage element interface 606, and one or more storage elements 608. The bus 610 can include one or more conductors that enable communication between components of the computing system 600. The processor 602 can include any type of conventional processor or microprocessor that interprets and executes programming instructions. The local memory 604 can include a random access memory (RAM) or another type of dynamic storage device that stores information and instructions for execution by the processor 602, and / or a read-only memory (ROM) or another type of static storage device that stores static information and instructions for use by the processor 602. The input interface 614 can include one or more conventional mechanisms that enable an operator or user to input information into the computing device 600, such as a keyboard 620, a mouse 630, a pen, a voice recognition and / or biometric mechanism, a camera, etc. The output interface 616 can include one or more conventional mechanisms that output information to an operator or user, such as a display 640, etc. The communication interface 612 can include a mechanism such as any transceiver, such as one or more Ethernet interfaces, that enables the computing system 600 to communicate with other devices and / or systems, such as a smartphone 650. The communication interface 612 of the computing system 600 can be connected to such another computing system by a local area network (LAN) or a wide area network (WAN), such as the Internet.The storage element interface 606 includes a storage interface, such as a Serial Advanced Technology Attachment (SATA) interface or a Small Computer System Interface (SCSI), for connecting a bus 610 to one or more local disks, such as one or more storage elements 608 like SATA disk drives, and can control reading and writing of data to and from those storage elements 608. Although the storage elements 608 are described as local disks, generally any other suitable computer-readable media can be used, such as removable magnetic disks, optical storage media like CD or DVD-ROM disks, solid state drives, flash memory cards, etc.

[0079] The present invention has been described with reference to specific embodiments, but it will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented with various changes and modifications without departing from its scope. Therefore, this embodiment should be considered exemplary in all respects and not restrictive, and the scope of the present invention is indicated by the appended claims rather than the above description. Thus, all changes within the meaning and equivalent scope of the claims are intended to be included therein. In other words, all changes, variations or equivalents falling within the scope of the underlying basic principles are covered, and the essential attributes thereof are considered to be claimed in this patent application. Further, it should be understood by the reader of this patent application that the terms "comprising" or "comprises" do not exclude other elements or steps, and that the terms "a" or "an" do not exclude a plurality, and that a single element such as a computer system, a processor or another integrated unit can perform the functions of a plurality of means recited in the claims. Any reference signs in the claims should not be construed as limiting the respective claims. Terms such as "first", "second", "third", "a", "b", "c", etc., when used in this specification or in the claims, are introduced to distinguish similar elements or steps and do not necessarily describe a continuous or chronological order. Similarly, terms such as "above", "below", "upper", "lower", etc., are introduced for illustrative purposes and do not necessarily indicate a relative position. Terms used in this way are interchangeable under appropriate circumstances, and it should be understood that embodiments of the present invention can operate in accordance with the present invention in other orders or orientations different from those described or illustrated above.

Claims

1. A computer-implemented method (100) for calibrating a digital image (110) of skin tissue, wherein the digital image includes a calibration marker (120) located on or near the skin tissue, the calibration marker including benchmark elements (121-131) characterized by at least one benchmark attribute value of at least one attribute type, and the computer-implemented method comprising: - a step (101) of detecting a calibration marker (220) in the digital image (110); - a step (102) of detecting benchmark elements (121-131) in a cropped portion (111) of the digital image including the calibration marker; - for each benchmark element (121-131), a step (103) of identifying at least one depicted attribute value (113) of the at least one attribute type based on pixels of the digital image located within each benchmark element; - a step (104) of calibrating the digital image by correcting a deviation between a benchmark attribute value (114) associated with each benchmark element and the depicted attribute value (113). A computer-implemented method characterized by comprising the above.

2. In the computer-implemented method according to Claim 1, the step (101) of detecting the calibration marker is executed by a machine learning model trained to detect a calibration marker in a digital image. A computer-implemented method characterized by this.

3. In the computer-implemented method according to any one of the preceding claims, the step (102) of detecting the benchmark elements is executed by a machine learning model trained to assign each pixel in the cropped portion of the digital image to the benchmark elements (121-131). A computer-implemented method characterized by this.

4. In the computer-implemented method according to any one of the preceding claims, one or more benchmark elements (121, 122, 123) are characterized by respective benchmark color values of a color type, and the step (104) of calibrating the digital image includes a step (203) of correcting the color values of the pixels in the digital image. A computer-implemented method characterized by this.

5. In the computer-implemented method according to Claim 4, The step (103) of identifying the described attribute value of at least one of the color types includes a step (201) of obtaining an average described color value (220) of pixels (213) of a digital image located within each of one or more benchmark elements (121, 122, 123), characterized by a computer-implemented method.

6. In the computer-implemented method according to claim 5, the method further includes a step (202) of obtaining a respective color deviation (222) between the average described color value (220) and a benchmark color value (221) associated with each of one or more benchmark elements (121, 122, 123), characterized by a computer-implemented method.

7. In the computer-implemented method according to any one of the preceding claims, at least four benchmark elements (124, 125, 126, 127) are characterized by a benchmark position value (314) of a position type, and the step (104) of calibrating the digital image includes a step (303) of adjusting the viewpoint of the digital image, characterized by a computer-implemented method.

8. In the computer-implemented method according to claim 7, the step (103) of identifying the described attribute value of at least one of the position types includes a step (301) of identifying the described positions (313) of each of at least four benchmark elements (124, 125, 126, 127), characterized by a computer-implemented method.

9. In the computer-implemented method according to claim 8, the method further includes a step (302) of determining a described polygon (341) defined by the described positions of each of at least four benchmark elements (124, 125, 126, 127) within the digital image, and the step (303) of adjusting the viewpoint of the digital image further includes a step of mapping pixels within the described polygon (341) to pixels within a benchmark polygon (321) defined by the benchmark position values of the at least four benchmark elements, characterized by a computer-implemented method.

10. In the computer-implemented method according to any one of the preceding claims, One or more benchmark elements (130, 131) define a reference axis (511) of the calibration marker (511), and the step of calibrating the digital image includes a step (502) of rotating the digital image based on the reference axis. A computer-implemented method characterized by that.

11. In the computer-implemented method according to claim 10, The method further includes a step (503) of obtaining an angle (512) between a reference axis (511) of the calibration marker (120) and a preset edge (513) of the digital image (510). A computer-implemented method characterized by that.

12. In the computer-implemented method according to claim 10, The reference axis (511) is characterized by a benchmark position on the calibration marker (120), and the rotation (505) of the digital image is further performed based on the depicted position of the reference axis. A computer-implemented method characterized by that.

13. In the computer-implemented method according to any one of the preceding claims, One or more benchmark elements (124, 125, 126, 127) are characterized by a benchmark surface area value (411) of a surface area type, and the computer-implemented method includes the number of pixels (410) located within each one or more benchmark elements and the benchmark surface area (411) of each one or more benchmark elements, and further includes a step (401) of obtaining a surface area associated with each pixel of the digital image. A computer-implemented method characterized by that.

14. In the computer-implemented method according to claim 12, The step of obtaining the surface area associated with the pixels located outside the one or more benchmark elements (124, 125, 126, 127) includes interpolating (402) and / or extrapolating (403) the surface area associated with the pixels located within the one or more benchmark elements. A computer-implemented method characterized by that.

15. A data processing system configured to execute the computer-implemented method according to any one of claims 1 to 14.

Images