Image processing

Abstract

Machine-readable instructions executed by the data processor for processing a captured image comprising a lesion to determine a boundary of the lesion, the instructions arranged to: display an image comprising the lesion being studied and comprising a skin portion adjacent to the lesion; determining a lesion skin reference hue in the image and determining an adjacent skin reference hue in the image, including selecting one or more representative pixels that do not belong to the lesion, the adjacent skin reference hue being a hue of skin adjacent to the lesion; analyzing pixel hue levels in the image to determine pixels of the image as lesion hue pixels or adjacent skin hue pixels; and determining, in the image, a boundary at which the diseased skin tone pixel intersects an adjacent skin tone pixel.

Description

Technical Field

[0001] The present invention relates generally to image processing and data acquisition. Background Art

[0002] It is well known that clinicians can use images of lesions to assess whether a lesion can be ruled out as potentially malignant. We have realized that improved edge detection can lead to more effective assessment of skin lesions. The differences between melanoma images and “non-melanoma” images can be very subtle. Many lesions do not have easily defined edges. Pigmented skin lesions are often similar in color to the surrounding skin. In addition, the shape of the lesions can be very irregular. A skin lesion can have an outer colored ring, then a darker edge, and yet another edge within the lesion. This makes visual judgment of edges subjective and can lead to wide variations in assessments of known image manipulations.

[0003] We devised an improved method for determining the boundaries of skin lesions. Summary of the invention

[0004] According to a first aspect of the invention, there are provided machine readable instructions for execution by a data processor for processing an image comprising a lesion to determine a boundary of the lesion, the instructions being arranged to perform at least the following steps:

[0005] displaying the image, the image including the lesion under investigation and including a portion of skin adjacent to the lesion;

[0006] Determining a reference skin tone of a lesion in the image, including selecting one or more representative pixels within the lesion;

[0007] Determining an adjacent skin reference hue in the image, including selecting one or more representative pixels that do not belong to the lesion, the adjacent skin reference hue being the hue of the skin adjacent to the lesion;

[0008] In the image, boundaries are determined where diseased skin tone pixels intersect adjacent skin tone pixels, wherein the image pixels are analyzed to determine tone levels.

[0009] A "lesion" includes a portion of the skin that has an abnormal growth or appearance compared to the surrounding skin.

[0010] An "adjacent skin reference tone" includes the tone of the skin adjacent to the lesion and / or the tone of the skin surrounding the lesion, and / or it may not substantially include the lesion. The tone reference may include a color, so different colors and / or variations of colors may be used.

[0011] The present invention may be viewed as enabling data acquisition with improved accuracy.

[0012] One or more actions may include performing image processing calculations.

[0013] Instructions may include:

[0014] Image display module / block,

[0015] at least one skin tone analysis module / block, and

[0016] Boundaries determine modules / blocks.

[0017] The step of analyzing the tonal level of the pixels may include starting with analyzing one or more pixels adjacent to the corresponding one or more representative pixels. One or both of the reference tones may include an average tonal value obtained from a plurality of pixels. The instructions may cause the tonal level analysis to be progressively further away from the corresponding one or more representative pixels.

[0018] The instructions may be configured to analyze pixels in the image individually and / or in groups and determine whether the pixels are lesion tone pixels or adjacent skin tone pixels. The analysis may include determining whether the detected tone level is within a corresponding reference tone range, or exceeds a tone reference value, or meets a reference tone value (possibly within a tolerance level). The calculated or predetermined tone reference range or reference value may be used for the analysis of adjacent skin tone and / or lesion tone, which may be determined at least in part by the skin lesion tone and / or adjacent skin tone determined from the image.

[0019] For analysis of adjacent skin tone pixels, the progression may be toward the center of the image. For lesion tone pixels, the progression may be away from the center of the image. These steps may be referred to as "filling" the image space, starting with the corresponding representative pixel or pixels, and assigning (and storing) a value or indicator indicating whether the pixel or group of pixels is a lesion skin tone or an adjacent skin tone.

[0020] The step of determining a reference skin tone of the lesion may include determining a darkest level of skin tone of the lesion (as shown in the image). The step of determining a darkest level of skin tone of the lesion may include identifying at least one pixel within the lesion region that is determined to be a pigment from a subset having substantially the darkest tone level of the lesion. The selected pixel may have the darkest tone or be one of a plurality of pixels in the region having substantially the same tone level, the substantially same tone level being the darkest tone level of the pixels within the region.

[0021] The step of determining a reference skin tone for a lesion may include determining a set of pixels for the lesion. The set of pixels may be located in a central region of the image or near a central region of the image. The step of determining a reference skin tone for a skin lesion may include identifying a lesion region of a predetermined pixel size or size. For example, the size of the region is 100x100 pixels, + / - 25x25 pixels. The region may occupy a substantially central portion of the image, or may overlap a central point of the image.

[0022] Determining an adjacent skin reference tone of skin adjacent to the lesion may include identifying at least one pixel within the image that is away from the lesion. The at least one pixel may be in a corner region of the image. The step may include identifying at least one pixel in more than one corner region of the image, or may include identifying at least one pixel in each of a plurality of corner regions of the image.

[0023] The lesioned skin tone may be darker than the adjacent skin tone, and the adjacent skin tone may be lighter than the lesioned skin tone.

[0024] The instructions may be arranged to generate an initial determined boundary of the lesion. The instructions may be arranged to determine a refined boundary with a higher precision than the initial boundary. The refined boundary may be generated at least in part using or based on the initial boundary. Generating the refined boundary may include analyzing the image with a higher precision.

[0025] The instructions may be used to generate a cropped and / or enlarged image that includes the lesion based on the location of the (initial) boundary in the image (which in turn should indicate the location of the lesion). A refined boundary may be determined based on the use of the cropped and / or enlarged image. The instructions may be used to determine the refined boundary using a higher resolution image (compared to the resolution of the image used to determine the initial boundary).

[0026] The instructions may be used to determine an initial boundary and / or refine a boundary using pixel tone value analysis in one or more of the ways set forth in the preceding paragraphs.

[0027] The instructions may be arranged to use a captured image having a lower resolution than the captured image to determine the initial boundary. The resolution of the version image may be less than fifty percent of the resolution of the captured image, or may be twenty to forty percent of the resolution of the captured image. The instructions may be arranged to generate a lower resolution version image than the resolution of the captured image.

[0028] Where an initial boundary of the lesion has been determined, instructions may be used to locate the initial boundary in the (original) captured image, or to locate the initial boundary in a specified version of the image having a higher resolution than the image used to determine the initial boundary (but lower than the resolution of the captured image). The higher resolution need not correspond to the resolution of the captured image. Locating the initial boundary in the higher resolution image may include the steps of scaling the coordinates of the initial boundary (by a factor determined by the relative size / resolution of the image used for the initial boundary and the higher resolution image) and then mapping the initial boundary onto the higher resolution image.

[0029] The instructions may then be used to generate a cropped image from the higher resolution image, including the initial boundary and including a margin of non-lesioned skin surrounding the lesion. The instructions may cause the lesion to occupy a majority of the cropped image, the lesion may occupy seventy-five to ninety percent of the cropped image. The lesion may be centered, or substantially centered, in the cropped image.

[0030] Instructions can be used to perform pixel tone analysis on the cropped image and determine refinement boundaries from there.

[0031] This (border determination phase) can be viewed as a two-stage process, where the lesion edge is first determined with a lower level of accuracy and then with a higher level of accuracy.

[0032] The boundary determination module / block may include an initial boundary submodule and a refined boundary submodule.

[0033] The instructions may include an image acquisition module / block for allowing and / or guiding the user to take a picture of the lesion and the skin portion surrounding the lesion.

[0034] The instructions include generating a centering graphic that may include two elongated linear features (such as lines, dashed lines, or solid lines) that are arranged substantially orthogonal to each other and that are arranged to be displayed to the user. The centering graphic may be referred to as a crosshair. The centering graphic is preferably arranged to be displayed in a field of view display of a camera function. The centering graphic may be overlaid or superimposed on the field of view of a camera function selected by the user, such as a view of skin with a lesion. The instructions may prompt and enable the user to use the camera function to take a photo of the skin with the lesion. In an image capture mode enabled by the instructions, a centering graphic may be presented to assist the user in taking a photo of the lesion and surrounding skin, including centering the lesion in the presented field of view. In image capture mode, an enlarged view may be displayed with the centering graphic superimposed on the view.

[0035] The instructions may direct or prompt the user to place the lesion substantially centered in the image display area. This may be required before starting to determine the lesion skin tone and adjacent skin tone. The instructions may cause the user to continue determining the boundaries of the lesion in a substantially automated manner once the user centers the image to be captured.

[0036] The instructions may be configured to include the step of generating an enlarged version of the image.

[0037] The instructions may be configured to output an image including the determined boundary and the lesion, the determined boundary being superimposed on the image of the lesion. The determined boundary may be a refined boundary. The instructions may prompt a user to view the image, and to provide input related to a user assessment of the perceived accuracy of the (computed and) displayed boundary, the (computed and) displayed boundary corresponding to the lesion boundary. The instructions may require affirmative user input to be classified as appropriate to perform one or more assessments. Otherwise, some or all of the boundary determination procedure may need to be re-performed.

[0038] The instructions may be embodied as a software product, which may be a software application or an application program. The software product may be used on / executed by a personal electronic device (PED), such as a smartphone or tablet. More generally, the instructions are suitable for execution by a computer comprising a data processor and a memory.

[0039] The instructions may be configured to include a graphical user interface.

[0040] According to another aspect of the present invention, instructions are provided which, when executed by a processor, determine the boundary of a lesion using a two-stage process, wherein a first initial (approximate) lesion boundary is determined using pixel tone analysis, which is then applied to a higher resolution version of the image to determine a refined lesion boundary using pixel tone analysis.

[0041] According to another aspect of the present invention, a user device is provided, which is loaded with the machine-readable instructions described in the first aspect or another aspect of the present invention.

[0042] The user device may be a personal electronic device (PED), such as a smartphone or tablet device, which includes a camera, a display screen, a data processor and a memory. Typically, such a device is portable.

[0043] As disclosed in the specification or drawings, any aspect of the invention may include one or more features, alone or in combination. The disclosure includes that a single feature or multiple features may be incorporated into any aspect of the invention, even though they are disclosed as connected or associated with other features, for example, in the context of a specific embodiment or sequence or combination of physical or method or process steps. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Various embodiments of the present invention will now be described, by way of example only, with reference to the following drawings, in which:

[0045] Figure 1 is a schematic diagram of a personal electronic device PED;

[0046] Figure 2 is an enhanced representation of the captured image of the lesion and surrounding skin;

[0047] Figure 3 is a view showing an image, which shows a close-up view of a portion of the image shown in the figure, including a centered crosshair;

[0048] Figure 4 is a graphical user interface that includes an image of the lesion and surrounding skin and a picture of the lesion in which the determined boundary is overlaid on the lesion; and

[0049] Figure 5 is a flow chart showing the sequence of processing steps performed when determining lesion margins. DETAILED DESCRIPTION

[0050] A new system for acquiring accurate edges of skin lesions is described. A user device configured with camera functionality, such as a smartphone or tablet device, executes a software application that includes various functional modules implemented. Figure 1 An example of such a device 100 is shown in FIG. 1 , which includes a touch screen 101 and user input buttons 102. In general, through displayed instructions, the user is guided to take a picture of the lesion and then can analyze the image to determine whether the malignancy of the lesion can be ruled out. As described in more detail below, the analysis includes determining the boundaries of the lesion.

[0051] To begin using the system, the user opens the software on their PED. The user is then first prompted to point the PED's camera at the lesion to be treated. The user (i.e., the person using the system) is not necessarily the person with the lesion.

[0052] During the image capture phase, the PED's screen displays the camera's field of view. The user's task is to take a picture of the lesion and surrounding skin. Figure 2 In the example shown, the lesion 5 is located on the forearm of a person. As shown, the captured image includes the lesion and the (lesion-free) skin surrounding the lesion. To help the system capture the best image, a centering guide is displayed, such as Figure 3 As shown. It shows a magnified image of the lesion. Text (not shown) displayed to the user prompts the user to center the lesion with reference to the crosshairs and to ensure that the lesion is within the circular boundary (the field of view outside the circular boundary is displayed as "gray" or partially blurred in the displayed camera field of view). Therefore, the user moves the PED to locate the lesion. Based on this, the user can take a photo. As shown Figure 2As shown, the lesion in the obtained image is substantially located in the center of the image frame.

[0053] After the image is captured, the software application begins processing the image and begins the edge determination phase. In the first part of this phase, the approximate boundaries of the lesion are determined as follows. For this initial boundary determination, a lower resolution version of the captured image is generated and used.

[0054] Select a sample of the pixel set from the center area of ​​the image frame. It can be a square of 100x100 pixels (such as Figure 2 ). The pixels contained in the set are analyzed to determine the darkest pixel, i.e., the pixel with the darkest hue, or more than one pixel in the set with the same darkest hue, and the hue levels of these pixels are stored. This can be referred to as the lesion hue reference level.

[0055] Afterwards, the surrounding skin tone level is determined. This can be achieved by selecting one or more pixels from each corner of the image frame. Figure 2 , this is depicted by icon 9 for ease of explanation. A magnifying glass icon 12 is shown depicting that the lower left corner region is being analyzed. A selection of pixels from each corner is analyzed to determine the brightest tone of the pixels. The brightest tone level determined is required to be brighter than a reference tone level determined for the lesion. The point at which the pixels are selected in each corner is predetermined, and is, for example, a predetermined distance from each corner along a diagonal line connecting opposite corners of the image frame. For example, one tenth of the pixels diagonally from each corner of a close-up image may be used to determine the corner region to be analyzed for determining a reference surrounding skin tone level. In the event that any corner of the image has artifacts other than skin, or in the event that any corner region does not include skin due to limb curvature or shadow casting, the corners are analyzed sequentially. It is assumed that there is at least one corner with brighter skin for the system to operate.

[0056] For the process of lesion edge determination, a difference in tone is required between the lesion and the surrounding skin in order to detect edge drops. For automatic edge drop detection, it is assumed that the lesion is darker than the surrounding skin.

[0057] Having determined the reference levels for each skin tone, the following image processing steps can be implemented.

[0058] The lesion boundaries are then initially (or approximately) determined. This is accomplished by analyzing the tonal level of each pixel in each lesion region and surrounding skin region relative to a reference tonal level. In this case, each pixel is determined to be either a pixel that depicts the surrounding skin or a pixel that depicts a portion of the lesion. The two regions must maintain contrast between their respective tonal minima. In other words, each region has a distinct range where the skin is much lighter than the lesion, and when the growth encounters a step change, this contrast produces a mask that represents the lesion and the surrounding skin becomes transparent. A specific starting pixel in each region, such as one of the pixels used to determine the reference tonal level, can be used as a seed for binary growth, where each neighboring pixel is determined to be either a lesion pixel or a surrounding skin pixel. Where they meet, a consistent edge is formed. A flood fill algorithm can be used from the center darkest pixel outward and from the skin tone pixels inward. A grayscale map is generated that is flood filled to eliminate any "holes". As a result, the edges of the lesion are approximately determined based on where the lesion pixels are adjacent to the surrounding skin pixels.

[0059] A cropped image of the lesion is now generated based on the location of the lesion edge just calculated, where the frame of the cropped image is very close to the edge. The cropped image is generated by scaling the coordinates of the initial boundary to a higher resolution version of the image, which can be the resolution of the image originally captured, or a higher resolution than the resolution of the image used to determine the initial boundary, and mapping the initially determined boundary onto the higher resolution image. The image is then cropped so that a large portion of the image includes the lesion, but still has a surrounding boundary of the surrounding skin.

[0060] The second stage of boundary determination is now implemented using the cropped image. The stored reference tonal levels are used again in the same manner as described above for determining the initial boundary, but at a more detailed level (since a higher resolution image is now used). As a result, the boundary of the lesion is determined with a high accuracy level of one pixel.

[0061] The two-stage process described advantageously provides very efficient image processing. For uncropped high-resolution images from a camera, "seed growing" can be relatively slow (minutes rather than seconds). By starting with a lower resolution image, the approximate boundary is first determined, which allows the high-resolution image to be cropped close to the edge of the lesion, thereby significantly reducing the seed growing processing time.

[0062] Figure 4 Shows a refined, higher precision boundary displayed to the user (this figure shows the same Figure 2 and Figure 315). The original image is shown, as well as a cropped image showing boundary 15. The user is prompted to provide input indicating whether the calculated boundary correctly encloses the lesion. If the user provides an affirmative answer, the boundary is determined to be sufficiently accurate for any subsequent analysis thereof. This analysis, which is not part of the present disclosure, may include calculating a measure of boundary irregularity, which, broadly speaking, can provide a measure of the likelihood that the lesion is malignant. If the analysis exceeds a threshold, indicating that the lesion is not malignant, it can be reported. Based on the analyzed features of the determined boundary, it can be determined whether malignancy can be ruled out. Otherwise, the user is advised to seek advice from a clinician. Further analysis performed on the determined boundary may also include (in addition to or as an alternative to boundary irregularity) analysis of the skin (and one or more features thereof) contained within the accurately determined boundary. Analysis of color, texture and / or variability thereof may be included. AI can be beneficially used in whole or in part for such further analysis. The further analysis may include diagnostic analysis.

[0063] In the above described embodiments, a deterministic approach is advantageously implemented. This means that if the process is repeated for the same lesion, the same boundary is determined. This is different from known edge determination methods, which may determine different edges on different instances of processing the same lesion.

[0064] Figure 5 is a flow chart summarizing the main steps implemented by the software.

Claims

1. Machine readable instructions for execution by a data processor for processing a captured image including a lesion to determine a boundary of the lesion, the instructions being arranged to perform at least the following steps: displaying the image, the image including the lesion under investigation and including a portion of skin adjacent to the lesion; Determining a reference skin tone of a lesion in the image, including selecting one or more representative pixels within the lesion; Determining an adjacent skin reference hue in the image, including selecting one or more representative pixels that do not belong to the lesion, the adjacent skin reference hue being the hue of the skin adjacent to the lesion; Analyzing the tonal levels of pixels in the image individually and / or in groups to determine the pixels of the image as lesion tonal pixels or adjacent skin tonal pixels, wherein the analysis is performed on each of the one or more selected representative pixels that are progressively away from the lesion and each of the one or more selected representative pixels that are not part of the lesion, respectively; as well as In the image, boundaries where diseased skin tone pixels intersect adjacent skin tone pixels are determined.

2. The instruction according to claim 1, wherein: The instructions are arranged to determine a darkest level of diseased skin tone.

3. The instruction according to claim 2, wherein: The instructions are arranged to determine a darkest level of lesion skin tone by identifying at least one pixel within a lesion region, the at least one pixel being determined to be from a subset having substantially the darkest tone level for the lesion.

4. The instruction according to claim 3, wherein: The set of pixels is located in or close to the central area of ​​the image.

5. An instruction as claimed in any preceding claim, wherein: The instructions are arranged to determine an adjacent skin reference tone of skin adjacent to the lesion by identifying at least one pixel within the image that is distal to the lesion.

6. The instruction according to claim 5, wherein: The at least one pixel is analyzed at each of a plurality of locations in the image.

7. Instructions according to any one of the preceding claims, wherein: The instructions are arranged to generate an initial determined boundary of the lesion.

8. The instruction according to claim 7, wherein: The instructions are configured to analyze the tones of pixels in the image sequentially starting from at least two different locations, wherein at least one location is within the lesion and one location is outside the lesion.

9. An instruction as claimed in claim 7 or 8, wherein: The initial boundary is determined as the location where a diseased skin tone pixel is adjacent to a neighboring skin tone pixel.

10. An instruction as claimed in any one of claims 7 to 9, wherein: The instructions are configured to determine a refined lesion boundary having a higher precision than the initial boundary.

11. The instruction of claim 10, wherein: The instructions are configured to determine the refined lesion boundary using a higher resolution version of the image having a higher resolution than a resolution of the image used to determine the initial boundary and using the initial boundary.

12. The instruction of claim 11, wherein: The instructions are for mapping the initial boundary onto the higher resolution image.

13. The instruction of claim 12, wherein: The instructions are configured to generate a cropped version of the image that includes the lesion.

14. The instruction of claim 13, wherein: The instructions are configured to generate the cropped image once the lesion has been mapped onto the higher resolution image.

15. An instruction as claimed in claim 13 or 14, wherein: The cropped version image is generated by using the position of the initial boundary.

16. An instruction as claimed in any one of claims 11 to 15, wherein: The instructions are for generating the cropped image comprising a primary portion depicting the lesion and a secondary portion depicting skin surrounding the lesion.

17. An instruction as claimed in any one of claims 11 to 16, wherein: The instructions are for performing pixel tone analysis on the cropped image and determining the thinning boundary therefrom.

18. An instruction as claimed in any preceding claim, wherein: The instructions are configured to perform a two-stage process to determine the boundary, wherein a first stage includes computing an initial boundary at a first image resolution and then computing a refined boundary at a second image resolution, wherein the second image resolution is higher than the first image resolution.

19. An instruction as claimed in any preceding claim, wherein: The instructions are configured to generate a centering graphic in a graphical user interface to guide a user to place the lesion in the center of an image to be captured.

20. The instructions of claim 19, wherein: The instructions are arranged to provide the centered graphic overlaid on an enlarged portion of the camera's field of view.

21. An instruction as claimed in any preceding claim, wherein: The instructions are configured to generate an image output including a determined lesion boundary and the lesion, the determined lesion boundary displayed superimposed on the image of the lesion.

22. The instructions of claim 21, wherein: The instructions are configured to prompt a user to view the image and to provide input related to the assessment of the perceptual accuracy of the boundary.

23. Instructions according to any one of the preceding claims, wherein: The instructions are arranged to generate a lower resolution version of the captured image, the lower resolution image being used to determine an initial lesion boundary.

24. A user device loaded with the machine-readable instructions of any one of claims 1 to 23.

Images