Method for generating a differential marker on a representation of a portion of the human body

Abstract

The invention relates to a method for generating at least one differential marker of the presence of a skin singularity of a human body, said method comprising: ■ acquiring first and second sets of dermoscopic images of singularities of the skin of a human body of a first patient at first and second dates, respectively; ■ generating first and second representations of a first image of a part of the human body and of first symbol and second symbols, respectively, which are superimposed on the first image of each representation at a position in a first reference frame of the first image, said geometry and / or said colour of the second symbol being different from the geometry and / or the colour of the first symbol when the second class of the dermoscopic image of the second set is different from the first class of the dermoscopic image of the first set.

Description

[0001] Method for generating a differential marker on a representation of a portion of the human body.

[0002] Field of invention

[0003] The field of the invention relates to that of computer-implemented methods for generating markers on a representation of the human body. The field of the invention relates more particularly to systems and methods for assisting a doctor such as a dermatologist in the analysis of skin singularities on the surface of the body.

[0004] State of the art

[0005] Currently, dermatologists analyze skin peculiarities by examining the surface of a patient's skin with the naked eye.

[0006] Current imaging devices offer practitioners limited choices. Most commonly, a dermatologist can access dermoscopic images at 10x to 30x magnification of a few lesions, taken individually using a hand-held dermatoscope such as a "gun." Alternatively, macroscopic images of a body can be accessed using whole-body imaging systems, such as a "booth," which allow for overall body acquisition.

[0007] These two techniques can be used in combination, but at best and at the cost of too much time, only allow for obtaining a macroscopic image of the entire body, with some lesions in dermoscopy by manual association of dermoscopic images to an area of ​​the body.

[0008] As a result, there are no full-body imaging or skin maps that allow access to dermoscopy images at any point.

[0009] Some existing systems allow for taking photos of the skin at different resolutions. These systems allow for macroscopic observation of moles, for example. However, it is important to take images at different times, ideally in dermoscopy, to monitor the evolution of a singularity over time. The doctor must therefore save the images, associate them with an area of ​​the body to be able to compare them with the correct images later, and finally orient and display them in the same way to compare the images with each other during an examination.

[0010] However, to date, there is no process or system that addresses all of these issues.

[0011] Summary of the invention

[0012] According to a first aspect, the invention relates to a computer-implemented method for generating at least one differential marker of the presence of a cutaneous singularity of a human body, said method comprising:

[0013] ■ Reception on a first date of at least a first image of all or part of the human body, called first body part, of a first individual allowing the display of a dermoscopic image extracted from said first image with a dermoscopic resolution, said first image comprising a plurality of cutaneous singularities of the skin of said body, each singularity each having coordinates in a first reference frame associated with said first image and being associated with a first date and at least a first value of a first descriptor;

[0014] ■ Reception at a second date of at least one second image of the same first part of the human body of the first individual with a substantially identical resolution, said second image comprising a plurality of cutaneous singularities of the skin of said body, each singularity each having coordinates in a first reference frame associated with said second image and being associated with a second date and at least one second value of the first descriptor;

[0015] ■ Generation of a first representation comprising the first image and at least one first symbol associated with a first singularity located at a first position of said first image of the first reference point, said at least one first symbol being superimposed on the first image at the first position, said first symbol having a first geometry and / or a first color generated as a function of at least the first value of the first descriptor considered at the first date;

[0016] ■ Generation of a second representation in the vicinity of the first representation comprising the second image and at least one second symbol associated with the first singularity, said second symbol having a second geometry and / or a second color, said at least one second symbol being superimposed on the second image at the first position, said second geometry and / or said second color being different from the first geometry and / or the first color thus defining a differential marker, when the distance calculated between a first value of the first descriptor calculated at the first date and a second value of the first descriptor calculated at the second date is greater than a predefined threshold.

[0017] One advantage is that it provides a simple tool for comparing two skin images acquired on different dates while benefiting from easily accessible tools for visualizing areas of interest and allowing the evaluation of the evolution of a situation.

[0018] According to one embodiment, the method comprises generating a graph comprising a set of nodes, said nodes corresponding to singularities, each node comprising attributes, including a position of the singularity and at least one value of a descriptor.

[0019] One advantage is that it allows the two images to be merged according to the same reference point in order to allow a similar display of the two parts of the human body at two different dates.

[0020] According to one embodiment, the two images of each representation are oriented and aligned with each other by means of a step of comparing and / or merging the two graphs and minimizing the error of the position difference of the nodes between them.

[0021] One advantage is to facilitate the reading and analysis of a singularity by offering a better visualization context favoring the detection of differences from one image to another.

[0022] According to one embodiment, at least one characteristic vector is calculated at each node of the first graph and of the second graph by a machine learning model, said model receiving as input an image of a singularity and generating as output a characteristic vector of the similarity of said image.

[0023] According to one embodiment, the comparison step implements the optimization of a cost function for calculating a distance between two graphs taking into consideration:

[0024] • a first distance between the nodes of the first graph and the nodes of the second graph, said first distance using a geometric metric making it possible to calculate a distance between points in space,

[0025] • a second distance between the nodes of the first graph and the nodes of the second graph, said second distance using a metric making it possible to calculate a distance between characteristic vectors.

[0026] According to one embodiment, the optimization of the cost function of the distance between the two graphs makes it possible to apply a transformation to each node of a first graph to make it correspond to a node of the second graph.

[0027] According to one embodiment, the optimization of the cost function of the distance between the two graphs makes it possible to apply a non-rigid transformation.

[0028] According to one embodiment, each graph comprises between 5 and 600 nodes. And more particularly, this range of nodes for certain individuals having characteristic skin is between 50 and 500 nodes.

[0029] According to another aspect, the invention relates to a computer-implemented method of linking two images each having a dermoscopic resolution of all or part of a human body and being received on two different dates, said method comprising:

[0030] ■ Reception on a first date of at least a first image of all or part of the human body, called first body part, of a first individual allowing the display of a dermoscopic image extracted from said first image with a dermoscopic resolution, said first image comprising a plurality of cutaneous singularities of the skin of said body, each singularity each having coordinates in a first reference frame associated with said first image and being associated with a first date and at least a first value of a first descriptor, each singularity located in the first image defining a node of a first graph;

[0031] ■ Reception at a second date of at least one second image of the same first part of the human body of the first individual with a substantially identical resolution, said second image comprising a plurality of cutaneous singularities of the skin of said body, each singularity each having coordinates in a first reference frame associated with said second image and being associated with a second date and at least one second value of the first descriptor, each singularity located in the second image defining a node of a second graph;

[0032] ■ Generation of a first representation comprising the first image;

[0033] ■ Calculation of a correspondence vector from the comparison of a plurality of positions of singularities and at least one value of at least one descriptor of said singularities of the first graph and of the second graph;

[0034] ■ Generation of a second representation in the vicinity of the first representation comprising the second image, said second representation being generated so that the first and second images are oriented according to the same reference frame or are displayed according to the same dimension scale.

[0035] This aspect of the invention is combined with all embodiments of the other aspects.

[0036] According to one embodiment, each singularity of the first image is associated with a plurality of descriptors comprising at least one descriptor from the following list:

[0037] ■ A contrast value with respect to a representative value of an average color considered in the vicinity of the skin singularity; ■ A given class of a classifier of an output of a neural network having been trained with dermoscopic images of skin singularities;

[0038] ■ A characterization of a geometric data of the shape.

[0039] ■ A score corresponding to a scalar value or a numerical value obtained by the implementation of an algorithm processing as input an image extracted from the first image,

[0040] ■ A score obtained by calculating different values ​​of singularity descriptors considered in the vicinity of a given singularity.

[0041] One advantage is that it allows the identification of a category or value of a singularity, making it easier to associate a type of symbol according to the category of the singularity considered.

[0042] According to one embodiment, when a class is associated with a singularity after acquired images of the skin are provided to a neural network configured to deliver a classification of said provided images, at least one class is included among the following list of classes:

[0043] ■ a class relating to the geometry of the periphery of the singularity of a given dermoscopic photo,

[0044] ■ a class relating to the characterization of a geometry of the periphery of the singularity of a given dermoscopic photo with respect to a plurality of characterizations of geometries of peripheries of singularities of other dermoscopic photos considered in the vicinity of the given dermoscopic photo;

[0045] ■ a class relating to the color of a singularity;

[0046] ■ a class relating to the asymmetry of the geometry of the periphery of the singularity of a given dermoscopic image,

[0047] ■ a class relating to the diameter of the geometry of the periphery of the singularity of a given dermoscopic image, when said singularity has a substantially circular shape,

[0048] ■ a class relating to the area on which the singularity is present on the human body.

[0049] An advantage is to inherit the classes of a classifier of a neural network at the singularity level and therefore also a position in the first image. According to one embodiment, an evolution criterion is calculated quantifying the evolution of a descriptor of a singularity between two images of two acquisitions carried out on two different dates.

[0050] One advantage is that it allows attention to be drawn to changes or developments in singularity over time.

[0051] According to one embodiment, an evolution criterion is calculated from a distance defined between a first value of a descriptor of a first node of a first graph acquired at a first date and a second value of a descriptor of a second node of a second graph acquired at a second date, each graph being generated from a first image, respectively from a second image, said images corresponding to a body of the same individual and the first node and the second node having the same position within the first and second images.

[0052] One advantage is that it quickly allows differential calculations between two singularities of two images taken at two different dates whose positions are known in two different frames of reference and in the frame of one and the other. Indeed, the use of graphs makes it possible to corroborate the positions of singularities very easily, in particular from position correspondences and similarities of descriptors. It is thus much simpler to make the two frames of reference of the two received images coincide in order to facilitate comparisons of descriptor values ​​of a singularity that has evolved over time.

[0053] According to one embodiment, each singularity has a calculated position in the first image or the second image which corresponds to a geometric point characteristic of a shape characteristic of the geometry of said singularity. It may be an oval, an ellipse, a circle, a rectangle or a triangle.

[0054] According to one embodiment, the color and / or geometry of a symbol is / are selected according to:

[0055] ■ a criterion for membership of a singularity to at least one class of the classifier;

[0056] ■ a value of a descriptor exceeding a threshold value;

[0057] ■ the value of an evolution criterion of a descriptor of a singularity calculated between two first images acquired at two dates. An advantage is to allow a wide configuration of representation of the different criteria of demarcation of a given singularity or of an evolution of a singularity in time.

[0058] According to one embodiment, the shape of the symbol is a simple geometric shape such as a circle, a triangle, a square, a rectangle or a cross or star shape.

[0059] According to one embodiment, the color of the symbol is generated according to a color gradient associated with the evolution over time of a value of a descriptor of a singularity according to a predefined scale of values.

[0060] According to one embodiment, a third symbol is generated according to a given color and / or shape when a singularity is present in a first image acquired at a given position for the first time, said color or said shape of the third symbol making it possible to distinguish said symbol from another symbol to indicate the new appearance of said singularity.

[0061] One advantage is to represent the occurrence of a singularity that has not been identified in previous image acquisitions.

[0062] According to one embodiment, a user interaction with at least one displayed symbol makes it possible to generate a first digital instruction aimed at displaying at least one dermoscopic image in a display window, said displayed dermoscopic image corresponding to an image extracted from the first image associated with the position at which the symbol is displayed on the first image.

[0063] One advantage is that it can directly exploit the overall image of a patient's body without having to "glue" or process images taken by another device onto a macroscopic image.

[0064] According to one embodiment, a second digital instruction generated by a user action makes it possible to display two dermoscopic images side by side extracted respectively from a first image and a second image, said two dermoscopic images making it possible to display the singularities of the same position on the body according to the same resolution and according to the same scale of dimensions.

[0065] One advantage is to benefit from the exploitation of the graphs associated with the first image and the second image. The graphs are easily manipulated and allow for simple adjustments between the first and second images because only the descriptors and the positions of the nodes are considered.

[0066] According to one embodiment, a second digital instruction generated by a user action makes it possible to display two dermoscopic images side by side, said two dermoscopic images making it possible to display the singularities having the same position on the body in the same orientation.

[0067] According to one embodiment, the first image and the second image are 2D images of a part of the human body. One advantage is to represent parts of the body according to a plan view. This view allows better manipulation of the displayed image, in particular rotations and zooms.

[0068] According to one embodiment, the first image and the second image are 3D images of a part of the human body. An advantage is to better represent a patient's body and quickly visualize an area of ​​interest.

[0069] According to one embodiment, a digital command allowing to move, zoom, or select an area of ​​interest of the first image of the first representation automatically causes the generation of an identical digital command on an equivalent area of ​​interest of the second image of the second representation. An advantage is to coordinate the representations of the two images during an examination of the human body represented on a computer with the taking into account of a temporal evolution criterion.

[0070] According to one embodiment, a first digital command makes it possible to orient a three-dimensional digital avatar of the human body so as to display a portion of the body selectable by a second digital command producing the display of a first image, said first image representing a plurality of singularities of the skin of said displayed portion, a set of symbols being represented with a geometry and a color dependent on at least one value of a descriptor of the singularity.

[0071] According to one embodiment, a first digital command makes it possible to orient a three-dimensional digital avatar of an individual's body so as to display a portion of the body, a third digital command making it possible to enlarge said portion of the body displayed on an area of ​​interest, said area of ​​interest displaying a plurality of markers each having a position on the surface of the human body in a reference frame associated with the digital avatar, each marker being associated with a singularity of the human body, a fifth digital command making it possible to select said marker to display a dermoscopic image extracted from the first image, said extracted image being defined around the position of the selected marker.

[0072] According to one embodiment, the dermoscopic images are acquired by an image-taking device configured to acquire a plurality of images of the skin of a human body of an individual and to assign to each image a position on a 3D model representing the body of said individual.

[0073] According to one embodiment, the 3D model representing the individual's body corresponds to the three-dimensional representation allowing navigation on different portions of the body.

[0074] According to one embodiment, the method comprises a step of detecting a set of singularities on the surface of the body comprising the execution of a neural network processing as input images of the skin acquired by an image acquisition device and generating as output a classification of said image, each of the images being indexed according to a position on the surface of a 3D body model reconstructed from a generation of a depth map.

[0075] According to another aspect, the invention relates to a system comprising an electronic terminal comprising a display for generating the images produced by the method of the invention and a data exchange interface for receiving images acquired by an image acquisition device or another computer or another memory when said received images have been previously processed following their acquisition by an image-taking device.

[0076] Another object of the invention relates to a method for performing an action on an image of all or part of the human body having a dermoscopic resolution so as to visualize on the one hand images of cutaneous singularities of the skin at a dermoscopic scale and on the other hand an image of all or part of the body at a macroscopic scale. An advantage of the invention is to generate maps of the skin making it possible to provide access to given areas of the skin by magnification functions up to the dermoscopic level.

[0077] One advantage is that it allows access to any image acquired and associated with a point in a representation of the body, in particular points corresponding to areas of virgin skin, or points corresponding to pigmented or non-pigmented lesions.

[0078] Brief description of the figures

[0079] Other characteristics and advantages of the invention will emerge on reading the detailed description which follows, with reference to the appended figures, which illustrate:

[0080] ■ Figure 1: a three-dimensional representation of a human body allowing navigation to select a body portion;

[0081] ■ Figure 2: two representations of the same image representing the torso of a human body at two different dates, each image comprising a set of singularities of the human body and a plurality of symbols making it possible to discriminate certain singularities from other singularities;

[0082] ■ Figure 3: a representation of an image of a portion of the human body allowing access to a dermoscopic photograph of a singularity;

[0083] ■ Figure 4: a representation of an image of a portion of the human body allowing access to two dermoscopic photographs of a singularity obtained on two different dates.

[0084] ■ Figure 5: an example representation of a system of the invention.

[0085] The term "dermoscopic image" means an image acquired by an optical device making it possible to form an image of a portion of the skin in close-up. In the remainder of the description, a dermoscopic image corresponds in the broad sense to an image of an area of ​​the skin having a magnification of the size of the imaged area. The magnification therefore comprises an operation aimed at representing an area with a scale larger than the scale corresponding to the actual size. The invention refers more particularly to dermoscopic images having magnifications of the order of 10x to 30x of the actual size of the area of ​​the skin considered, that is to say between ten times and thirty times the size of the actual area. According to one example, the dermoscopic image can be associated with a given resolution or greater than a given threshold.

[0086] Finally, according to another example, the dermoscopic image can refer to an image acquired by a device or instrument designed to image areas of the skin. This can be a dermatoscope which exists in different variants:

[0087] ■ Contact or non-contact equipment;

[0088] ■ Equipment projecting polarized or non-polarized light;

[0089] ■ Equipment comprising optics enabling the acquisition of digital or numerical images,

[0090] ■ Conventional equipment including, for example, a magnifying glass without acquisition.

[0091] According to another example, the method of the invention can be carried out from images acquired from certain devices with a magnification of up to 400x, or 400 times.

[0092] Any image with dermoscopic resolution is called an image, regardless of its size, which can be zoomed or magnified so as to display images at a dermoscopic scale, that is, an image with a magnification between 10x or 30x of the actual size of the singularity represented on the body of an individual.

[0093] A body model faithful to an individual's body is a body model resulting from a 3D scanning operation of an individual's body that includes images of the skin at each pixel of a three-dimensional representation. "Body" means the entire or almost the entire body, namely that certain parts of the body may be intentionally masked without declassifying the designation of body.

[0094] A skin singularity is defined as an area with a contrast with the average skin color in its vicinity, or a roughness of the human body located at a certain point on the body. Color or intensity thresholds can be defined to determine whether the area includes a singularity. In other cases, a singularity can be characterized by an area with a color or intensity gradient. These can be in the visible light range or in other parts of the spectrum such as the UV spectrum or the infrared spectrum, i.e. multispectral / hyperspectral.

[0095] Figure 1 illustrates a representation of an individual's body in 3D. This image may correspond to the first IMi image having a dermoscopic resolution and which can be zoomed directly to obtain an image of a SGi singularity at the dermoscopic scale. Alternatively, this representation allows, by means of a computer for example, to select an area of ​​the body such as the Zi area representing the arm of the body.

[0096] According to one example, a CR command allows a computer user to orient the 3D image of the body according to the view that he wishes to display and use. He can thus visualize, for example, the back or the torso of a body. To this end, a command makes it possible to rotate the body according to an axis of revolution here defined by the axis parallel to the axis along which the body extends according to its largest dimension. According to one example, other commands make it possible to orient the body around another axis, this may be a roll, pitch or yaw axis. According to one example, a translation can also be defined. If the 3D image is the first image received having a dermoscopic resolution, the latter can be manipulated so as to allow zooming on an image at the dermoscopic scale.

[0097] The user commands allow to generate digital instructions CONS2 allowing to move, modify, zoom, orient or even coordinate the display of the two images between them. Different digital commands CDN1 to move an area of ​​interest, CDN2 to orient the avatar according to at least one axis, CDN3 to magnify an area of ​​interest, CDN4 to select a marker positioned in the first image or the second image relative to a singularity to display an image extracted from the first image at a dermoscopic scale can result in generating different operations on the displayed image or on the two images displayed in each representation.

[0098] The selection of a marker of interest or a symbol of the image by a user makes it possible to generate a digital instruction CON1 interpretable by a computer and making it possible to generate a window in which a dermoscopic resolution image extracted from the first image or the second image is displayed at the dermoscopic scale.

[0099] According to an exemplary embodiment, a CDNo command makes it possible to enlarge an area of ​​the human body displayed on the screen. Thus, once a view has been chosen, for example that of the front face of an individual Ui , it is possible to zoom in on only one part of the body, such as the torso or the arm. Alternatively, it is possible to select an area of ​​the body in order to generate a representation in another window.

[0100] The representation can be a 2D or 3D view. According to the example in Figure 1, it is a three-dimensional representation, but a 2D representation could alternatively be chosen. This 2D representation can concern the entire body or a part of the body. In order to obtain a 2D representation of a 3D surface, a cut along demarcation lines makes it possible to generate a flat area of ​​a portion of the 3D surface for its representation on the screen.

[0101] The invention therefore comprises a first representation allowing navigation within the surface of the human body by operations of modification of the orientation, selection of zones or magnification, etc.

[0102] According to a first example, the representation of the human body such as that of figure 1, is an avatar independent of the faithful representation of the body of an individual, it is noted MODi.

[0103] According to a second example, the representation of the human body such as that of figure 1, is a faithful avatar of the body of an individual MODo which has been scanned from an image acquisition device. The avatar can be represented and oriented in a frame Ro linked to the body of the individual. When this representation is produced, it is based on the exploitation of a 3D model of a human body of which each point of the surface is indexed in a frame Ro linked to the body model. The invention is compatible with whole body imaging devices which can produce macroscopic images of the human body from which a faithful body model can be produced.

[0104] In both cases, a plurality of dermoscopic images is recorded in a memory in order to produce a single image IMi of all or part of the human body. The method of the invention can begin at the step of receiving the first image IMi. However, a step of detecting cutaneous singularities on the surface of the scanned body can be carried out beforehand. This operation aims to associate with each singularity a position of a MODo body model. Thus, when the 3D representation directly uses the 3D model, the positions of the singularities indexed on the MODo body model correspond to the positions of the markers generated on the surface of the human body represented and allowing access to the dermoscopic images.

[0105] The position of the singularities can also be indexed on a 2D image in a frame noted Ri.

[0106] When a 3D avatar is generated to represent a standard model of a human body, the markers generated on the surface of this representation are generated at positions corresponding to the positions indexed on the MODo body model that is faithful to the obtained scan of an individual's body. A coordinate transformation matrix can be used to generate coordinates from a faithful body model to a standard body model.

[0107] According to one embodiment, descriptors are associated with each singularity.

[0108] According to one embodiment, the coordinates of the points of a faithful 3D model of a human body of an individual are transposed within a 2D representation from a standard representation of a human body from a coordinate transformation matrix.

[0109] According to one embodiment, the system of the invention comprises an actuator such as a computer mouse and a pointer or a touch control allowing operations to be performed on the avatar such as a CR rotation noted in FIG. 1.

[0110] Figure 2 represents two representations of two images IMi, IM2 of the same part of a human body of the same individual U1 acquired on two different dates. The images IM 1 and IM2 are preferably displayed with the same scale and in the same orientation. They correspond to two 2D representations of a portion of the body such as for example the torso of an individual. According to one embodiment, the 2D representation of the portion of the human body is directly extracted from a faithful MODo body model of an individual's body resulting from a scanning operation of said individual. In the latter case, if the images are extracted from a body model that has evolved over time, for example because a long period separates the two acquisitions and the individual has followed a diet, then the two images IMi and IMi' are different.In other words, the Mo body model is different or the first image is different since skin pixels have changed or singularities may also have changed.

[0111] However, in the remainder of the description, it is considered that the images IMi and IM2 of each representation designate equivalent parts of the human body.

[0112] Each representation of a portion of the human body comprises an image IM1 and a plurality of symbols Si, S2 indicating that a plurality of singularities SG1 associated with the plurality of symbols belongs to a class of a given classifier. For example, it may be a mole, a pimple, an angioma, a scar, etc., or a particular type of each of its elements. Typically, there may be different classes of skin lesions and different classes of scars.

[0113] The method of the invention makes it possible to retrieve information from an already existing classifier in order to assign the class to the singularity detected and positioned in the image IM1 or IM2. For this purpose, a neural network can be configured according to a given training in order to produce as output a classifier making it possible to classify the images given as input to the network. This step as mentioned is preferably a step prior to the method of the invention.

[0114] Symbols can be geometric shapes such as circles, ovals, squares, triangles, hexagons, stars, etc.

[0115] Each symbol may also include a color, the color may also be automatically chosen to be associated with a class of a classifier. One advantage is to allow an intuitive display for a user of a representation of a portion of a human body indicating areas of interest while qualifying the area of ​​interest. The area of ​​interest in this case may refer to a singularity positioned at a position of the first image IM1 and / or the second image IM2. According to one embodiment, each singularity is represented by a marker 5 independent of the classification of the dermoscopic image associated with it.

[0116] Figure 2 represents within each image a plurality of markers 5 and a plurality of symbols Si, Si', S2, S2' on each representation.

[0117] In the case of Figure 2, the two representations correspond to two different acquisition dates DATE-i, DATE2 of the first and second images IM1, IM2 having a dermoscopic resolution of the human body. They can correspond to two visits of a patient spaced by a period of 6 months for example.

[0118] In the PRES1 representation, two symbols 14 and 16 are identifiable by a triangle which may indicate that the singularities are of the same type. These symbols do not seem to have changed class in the second PRES2 representation given that their geometry and color have not changed. The color is here interpreted by the shape of the dashes forming the outline of the geometric shape.

[0119] Note that the four symbols 10, 12, 13, and 15 are identifiable by a circle and may indicate that the singularities are of the same type and therefore of a different type than the singularities associated with symbols 14 and 16. Some of these symbols have changed in the second PRES2 representation. Indeed, symbol 12 has become symbol 12' and appears to have changed color. Remember that the lines forming the outline of the shape here represent the color. The shape of symbol 10 has changed geometry. Note that symbols 15, 13, and 11 have neither changed geometric shapes nor changed colors.

[0120] Thus, at a glance, a doctor can, for example, draw his attention to singularities allowing for a more in-depth examination.

[0121] In this case, some singularities may have malignancy due to an evolution of the outline of the shape of the singularity or due to the fact that they have been changed classes in the classifier or vice versa.

[0122] The method of the invention allows a saving of time in the examination of a patient and a reduction of human errors for example in subjects with numerous singularities. Figure 3 represents a PRESi representation of an image on which markers 5 and symbols Si, Si' are superimposed. Figure 3 also represents a dermoscopic image IMDi taken on a date DATE-i. This dermoscopic image can be displayed in a window following an action by a user on the first image IMi of the first PRESi representation. For example, a click from a mouse pointer activates the display of a window in which an IMDi dermoscopic image is displayed. This image makes it possible to represent a singularity associated with the symbol 11 of the image IMi. The dermoscopic image IMDi displayed is for example extracted from the image IMi according to a predefined size of a predefined framing around the position of the singularity.In one example, the framing takes into account the geometry of the singularity's contour so as to display the entire singularity. Thus, the size of the singularity's magnification can be adapted to a defined dimension of a framing.

[0123] Figure 4 illustrates the PRESi representation in which the image IMi is displayed following an action by an operator at the level of the symbol 12 superimposed on the image IMi. A first dermoscopic image IMDi is displayed in a first window and automatically a second dermoscopic image IMD2 extracted from a second image IM2 acquired at a date prior to DATE1, for example at DATEo is displayed in a second window in the vicinity of the first window. The display of the second window can be automatic when the shape or the color of the symbol is associated with a change in the singularity of the same position. It can also be a new symbol associated with a singularity that has not been previously classified and newly classified in a given class of the classifier during the last acquisition, that is to say at the most recent date.

[0124] Thus, the geometric or colorimetric characteristic of a symbol associated with a dermoscopic image of a given singularity can be affected by a class assigned to said dermoscopic image, or by a change of class assigned to said dermoscopic image, or by the assignment of a first class of the classifier to said dermoscopic image or by the result of a mathematical operation carried out on characteristics calculated from two singularities of the same position considered at two different dates. An advantage of the invention is to produce an output of a system by means of an interface proposing to display skin maps by part or of the entire body from the macroscopic level to the dermoscopic level with a magnification ranging from x10 to x30. To move from one view to another, a zoom function can be activated.

[0125] Figure 5 represents an exemplary embodiment of a system of the invention comprising a display 20 making it possible to display a first image and a second image produced by the method of the invention. The first image, like the second image, can be reconstructed from images acquired by an image acquisition device 6. In the case of Figure 5, it is a mobile robot arm 6 and controllable using a trajectory calculator. In this case, the user Ui is lying on a table 22. According to another embodiment, the acquisition device 6 can be a cabin comprising fixed or mobile optics in which an individual Ui positions himself, for example, in a standing position.

[0126] Figure 5 also represents another acquisition device 5 arranged in a room making it possible to image the surface of a patient's body. This device is associated with a reference point R'. This device can, for example, make it possible to image a macroscopic representation of the human body and the optical device arranged in the distal part of the robot arm is, for example, configured to image areas of the body with dermoscopic resolution.

[0127] The first image and the second image can be reconstructed by aggregating images acquired by a dermoscopic resolution image acquisition device. According to another method, a composite image is produced from the selection of the sharpest pixels of each acquired image of the human body. According to one embodiment, the images directly acquired by the image capture device can be transmitted to a local or remote computer to feed a neural network trained with images of the skin. An advantage is that it makes it possible to determine classes of the acquired images and to reassign them to singularities positioned in an image reconstructed from all of the images.

[0128] According to one embodiment, a singularity is identified when an image is classified into a given group of classes. For example, the group of classes forming the singularities may include the mole class, the scar class, and the carcinoma class. Other example classes may define the singularity class.

[0129] Each singularity image with dermoscopic resolution is indexed to a 3D model of an individual's human body. According to one embodiment, the position of the singularity is associated with this indexing. For example, a point on the surface formed by the shape of the singularity can be used to define the position of the singularity on the 3D model. Another example is the barycenter of the shape delimiting the singularity. According to another example, it is the center of the mean circle approaching the surface of the singularity.

[0130] Each singularity can have a position in the 2D image acquired by the optics and a position in the 3D image of the human body. An image of a singularity can be, according to the embodiments:

[0131] ■ a 2D image in the visible spectrum;

[0132] ■ a multispectral or hyperspectral image;

[0133] ■ a 3D image;

[0134] ■ an ultrasound image, confocal microscopy

[0135] ■ a biopsy image of the singularity;

[0136] ■ a combination of all the above options

[0137] Each singularity defines the nodes of a graph and for this purpose includes an identifier in the graph. According to one embodiment, the graph of singularities includes a plurality of nodes, each node being connected to its neighboring nodes according to a topological distance defined on the surface of the 3D model. Neighboring nodes can be considered to be geometrically close according to a defined metric, for example according to a Euclidean distance.

[0138] When the graph of singularities is defined with nodes as the position of singularities and edges as the link between neighboring nodes, it is possible to construct characteristic vectors for each node and / or each edge of the graph. These characteristic vectors can be defined as "embeddings" in English terminology and in the technical terminology applied to GNN, designating in English terminology "Graph Neural Network". Embeddings make it possible to encode information for each node and / or edge by encoding a characteristic data of the neighborhood of the node and / or edge.

[0139] We describe an embodiment, in which an embedding that we will call “characteristic vector” in the rest of the description, is calculated for each node.

[0140] One advantage of constructing a feature vector is to encode each node with its own data, such as a value that can form a node identifier or an attribute of the latter. The feature vector of a node can be calculated by a machine learning method in which the image of a singularity is provided as input and a feature vector is generated as output. The machine learning model is for example trained with a given pair of images and the implementation of two neural networks that train their coefficients according to the response of a cost function that we seek to minimize. This method makes it possible to train a machine learning model that will be used to generate a feature vector from an image. One advantage is to use the feature vector of a node as a point of comparison to quantify the similarity between two nodes.

[0141] According to one embodiment, the use of the trained model makes it possible to define a similarity function to calculate a distance between characteristic vectors between two nodes of the graph. The model is then trained so that the distance between two similar images is minimal when the images of the singularities are identical and is maximal when the images are very different.

[0142] The machine learning model is preferably trained with a large number of images of different singularities in order to construct discriminating characteristic vectors and to restore a high-performance distance function.

[0143] The inference of the model thus trained makes it possible to generate a given characteristic vector for an image of a singularity of a given node. This characteristic vector can then define an attribute or an identifier of the considered node of the singularity graph.

[0144] According to one embodiment, the characteristic vector of a node is calculated by taking into account the neighboring nodes. Thus, at each node, the characteristic vector encodes information, or even a distance according to a given metric, considered between two neighboring nodes. It is then possible to calculate values ​​of the characteristic vectors of a node from a plurality of neighboring nodes in order to calculate by iteration a characteristic vector locally restoring the level of similarity with the neighboring nodes. This method can be applied to each node of the singularity graph in order to generate a set of characteristic vectors of a graph. An advantage is to encode all the information relating to a graph such as its structure and its relations by considering characteristic vectors locally encoding the topology of the graph at a node.

[0145] One advantage of feature vectors, also called "embeddings", is to construct a digital representation of a graph that can subsequently be used to train machine learning algorithms. In the case of the present invention, feature vectors have the advantage of allowing the comparison of two singularity graphs of the same individual from whom two body models have been generated at two different dates. One of the objectives of the invention is to represent in the same reference frame the two body models and more particularly the dermoscopic quality images at each point of the body model.However, the body models taken at different dates of the same individual may have been acquired in different conditions, they may be disoriented from each other according to the 6 coordinates (3 rotations, 3 translations) or they may be different because the individual has changed, for example he may have gained or lost weight or have had an operation in the interval between the two dates thus modifying a portion of the graph.

[0146] Furthermore, an interest in using singularities as nodes of a graph to match two graphs from two body models of the same individual is to free oneself from the position of "classical" nodes which would be calculated following the generation of a 3D body model. Indeed, in the case of a point cloud generated from a scanning operation of a body defining a graph, the position of the nodes would be considered arbitrary during the reconstruction of the body model. Singularities make it possible to free oneself from an arbitrary position of the nodes considered on the surface of the body, in particular because they constitute a certain form of invariance in the individual, up to the new singularities. Thus, a set of characteristic vectors is calculated for each node of the graph of singularities.

[0147] According to a first example, the algorithm for calculating a characteristic vector only takes into consideration data from a singularity, such as its image.

[0148] According to another example, the algorithm for calculating characteristic vectors is initiated by calculating an initial characteristic vector for a chosen node, then the data or value defining the characteristic vector of the initial node is propagated to neighboring nodes. Each neighboring node in turn calculates its own characteristic vector and in turn propagates this value. This step can be carried out over several iterations in order to generate a set of characteristic vectors for all the nodes of the singularity graph. This technique is based on a passage of information between the nodes of the graph in order to calculate the set of characteristic vectors of the singularity graph. It is understood that when calculating a characteristic vector of a node, called the destination node, the latter takes into account data calculated by its neighbors in order to calculate its own characteristic vector.The operations performed to calculate a characteristic vector of a node from its own data and the data of neighboring nodes may include operations of the "average", "sum", "maximum", etc. type. The calculation of a characteristic vector for a node is for example calculated following several iterations, that is to say several passages of information between neighboring nodes of a given node so as to encode local information of the given node. The transmission of information is therefore repeated several times to allow each of the nodes to integrate the information of its neighbors and thus make it possible to capture more and more information on the structure of the graph.

[0149] When two singularity graphs are compared with each other, the comparison of characteristic vectors can be carried out by means of a mathematical function allowing the evaluation of the distance between singularity graphs.

[0150] To reduce computation, the comparison between two singularity graphs can be performed between two reduced graphs in order to converge quickly at the start and adjust later. For this, the dimension of the graphs can be reduced.

[0151] An advantage of this method of comparing singularity graphs is that it supports deformations between the two graphs. The comparison function preferentially implements one or more non-linear functions to perform the comparison operation, also called "matching".

[0152] According to one example, the method of the invention comprises an optimization step for modifying one of the two graphs to make it correspond to the other graph based on a cost function used both between the positions of each node of the graph and also between the characteristic vectors of each node of the graph. This method makes it possible to make the singularities of each graph correspond. This method makes it possible to be particularly efficient in supporting local changes between the two graphs while making close areas correspond locally to each other. The method is also robust to changes in singularities in one of the two graphs, the appearance of new singularities in one of the two graphs or even the absence of singularity in one of the two graphs.

[0153] This method is notably more efficient than a rigid transformation method because it supports changes in models over time.

[0154] The method of comparing graphs to make them match allows a global approach to the graph and a local approach to converge towards a superposition of the nodes, or of the majority of the nodes with new singularities appearing near and with singularities having disappeared near.

[0155] The global approach allows implementing a global cost function by considering all the nodes of the graph. The adjacency matrix can be used for this purpose. A distance, for example Euclidean, can be calculated between the two singularity graphs in order to optimize a cost function based on the topological distance between the nodes of two graphs. The function then seeks to minimize the distance function according to at least one criterion. According to one example, a first distance is defined between the nodes of each graph, for example, a Euclidean distance between points. A second distance can also be defined between the characteristic vectors of the nodes. Finally, a third distance can be defined between attributes or properties of the nodes of the graph. The attributes can correspond to descriptors of a node or to other data such as an annotation or a class.A plurality of distances can be defined to calculate a proximity between the nodes of two singularity graphs.

[0156] Note that when feature vectors are used to calculate a distance, it is possible to discriminate between the attributes used to calculate the feature vectors and those that did not contribute to calculating them. Thus, the second and third distances do not take into account the same criteria.

[0157] According to one embodiment, the distances can be more or less weighted with respect to each other in the algorithm for comparing and matching the two singularity graphs so as to optimize the calculations and obtain high correspondence accuracy. This weighting can for example be implemented to reinforce the Euclidean distance initially between the positions of the points with respect to another distance which will be reinforced in a second step to improve the convergence of the execution of the algorithm towards the solution or to improve the accuracy of the correspondence of the two graphs. According to another example, the weighting can make it possible to exploit in better proportions the distance between characteristic vectors of two graphs with respect to the descriptors of a singularity or the Euclidean distance between the positions of points of the two graphs.

[0158] In other words, the distance used in the latter case can be expressed as a combination of different distances between the nodes of the singularity graph, it can be expressed as: d = a di (Gi, G2) + b d2 (Gi, G2) + c ds (Gi, G2), where:

[0159] ■ di is a Euclidean distance to measure a topological distance on the 3D surface of the human body between nodes. Another distance can also be used than the Euclidean distance; ■ d2 any distance defined by a metric on a space, such as a Minkowski or Chebyshev distance, or any other type of distance such as a distance based on the "cosine similarity" function; d2 is for example used to calculate the distance between feature vectors (embeddings) of different nodes;

[0160] ■ ds any distance defined by a metric on a space, such as a Minkowski or Chebyshev distance, or any other type of distance such as a distance based on the “cosine similarity” function; ds allows for example to calculate the distance between descriptors of different nodes;

[0161] ■ a, b and c are coefficients;

[0162] ■ Gi: the graph of singularities generated at an instant ti;

[0163] ■ G2: the graph of singularities generated at time t2. d is the function that we are seeking to optimize.

[0164] As another example, the distance d can be expressed as a nonlinear function of di , d2, ds.

[0165] In one example, the vector of coefficients a, b and c is optimized during the graph matching operation.

[0166] For example, if feature vectors directly or indirectly encode descriptors, only the d2 distance can be used without having to use the ds distance.

[0167] According to one embodiment, additional data to that of the node positions, such as the class of the similarity image, can be used to define a distance ds. This distance ds can define in this case a discrete data 0 or 1 when the distance between two nodes is calculated. The distance d3 can be used with the distance di, for example, with or without the distance d2.

[0168] According to one embodiment, the optimization of the cost function of the distance between the two graphs makes it possible to define the parameters of a transformation of a first graph to make it correspond to a second graph. The transformation of the entire graph can correspond to the set of transformations of all the nodes of the similarity graph. The invention comprises the application of this calculated transformation in order to allow a display of similar portions of the two graphs and more particularly of the nodes of each graph, that is to say images of singularities. The display of the images of the singularities extracted from two nodes corresponding to each other in each of the graphs is preferably carried out side by side.

[0169] According to an exemplary embodiment, the pairs of corresponding nodes of each of the two graphs are associated. The association can be carried out by means of an index, a correspondence table, a database or even a data file. The node association data can then be used to represent nodes associated with each other.

[0170] According to one embodiment, the graph matching algorithm comprises an initialization phase, which comprises an initial registration of the graphs. This initial registration of the graphs can be carried out initially by a rigid transformation which aims to optimize a distance criterion between the nodes making it possible to bring the graphs closer together in space. According to one example, this initialization phase can be used to homogenize the dimensions of the graphs. According to another example, this initialization phase comprises the reduction of one graph so that it is topologically contained in the other graph. The algorithm proceeds by iteration to a homothetic transformation while optimizing the distance function.

[0171] According to an exemplary embodiment, a similarity graph comprises between 50 and 600 nodes. According to one embodiment, the similarity graph comprises between 150 and 500 nodes. The dimension of the node generally depends on the individual. However, one advantage is that the dimension of the graph allows robust algorithms to be applied while having relatively short calculation times given that the dimension of the graph remains small.

Claims

CLAIMS A computer-implemented method for generating at least one differential marker (Si, S2) of the presence of a cutaneous singularity on a human body, said method comprising: ■ Reception at a first date (DATE1) of at least one first image (IM1) of all or part of the human body, called first part, of a first individual (Ui) allowing the display of a dermoscopic image (IMD1) extracted from said first image (IM1) with a dermoscopic resolution, said first image (IM1) comprising a plurality of cutaneous singularities (SGi) of the skin of said body, each singularity (SGi) having each of the coordinates in a first frame (Ri) associated with said first image (IM1) and being associated with a first date (DATE1) and with at least one first value (V1) of a first descriptor (D1), each singularity located in the first image defining a node of a first graph, each node comprising attributes including a position of the singularity and at least one value of a descriptor (Di); ■ Reception at a second date (DATE2) of at least one second image (IM2) of the same first part (IM1) of the human body of the first individual (Ui) with substantially identical resolution, said second image (IM2) comprising a plurality of cutaneous singularities (SGi) of the skin of said body, each singularity (SGi) having coordinates in a second frame (R2) associated with said second image (IM2) and being associated with a second date (DATE2) and with at least one second value (V2) of the first descriptor (D1), each singularity located in the second image defining a node of a second graph, each node comprising attributes including a position of the singularity and at least one value of a descriptor (Di); ■ Generation of a first representation (PRES1) including the first image (IM1) and at least one first symbol (Si) associated with a first singularity (SGi) located at a first position of said first image (IMi) of the first frame (Ri), said at least first symbol (Si) being superimposed on the first image (IMi) at the first position, said first symbol (Si) having a first geometry (GEOi) and / or a first color (COLi) generated as a function of at least the first value of the first descriptor (Di) considered at the first date (DATE-i); ■ Generation of a second representation (PRES2) in the vicinity of the first representation (PRES1) comprising the second image (IM2) and at least one second symbol (S2) associated with the first singularity (SG1), said second symbol (S2) having a second geometry (GEO2) and / or a second color (COL2), said at least second symbol (S2) being superimposed on the second image (IM2) at the first position, said second geometry (GEO2) and / or said second color (COL2) being different from the first geometry (GEO1) and / or the first color (COL2) thus defining a differential marker, when the calculated distance between a first value (V1) of the first descriptor (D1) calculated at the first date (DATE-i) and a second value (V2) of the first descriptor (D1) calculated at the second date (DATE2) is greater than a predefined threshold, ■ The two images (IM1, IM2) of each representation (PRES1, PRES2) are oriented and aligned with each other by means of a step comparing the two graphs and minimizing the error in the positional difference between the nodes. A method according to claim 1, characterized in that at least one characteristic vector is calculated at each node of the first graph and the second graph by a machine learning model, said model receiving as input an image of a singularity and generating as output a characteristic vector of the similarity of said image. A method according to claim 2, characterized in that the comparison step implements the optimization of a cost function for calculating the distance between two graphs, taking into account: ■ a first distance between the nodes of the first graph and the nodes of the second graph, said first distance using a geometric metric allowing the calculation of a distance between points in space, ■ a second distance between the nodes of the first graph and the nodes of the second graph, said second distance using a metric for calculating a distance between characteristic vectors. A method according to any one of claims 2 to 3, characterized in that the optimization of the cost function of the distance between the two graphs allows a transformation to be applied to each node of a first graph to make it correspond to a node of the second graph. A method according to any one of claims 3 to 4, characterized in that the optimization of the cost function of the distance between the two graphs allows a non-rigid transformation to be applied. A method according to any one of claims 1 to 4, characterized in that each graph comprises between 50 and 600 nodes.Method according to claim 1, characterized in that each singularity of the first image (IMi) and / or of the second image (IM2) is associated with a plurality of descriptors (Di) comprising at least one descriptor from the following list:. ■ A contrast value relative to a representative value of an average color considered in the vicinity of the skin singularity; ■ A given class of a classifier of an output of a neural network having been trained with dermoscopic images of skin singularities; ■ A characterization of a geometric datum of the shape, ■ A score corresponding to a scalar value or a numerical value obtained by implementing an algorithm that processes as input an image extracted from the first or second image, ■ A score obtained by calculating different values ​​of singularity descriptors considered in the neighborhood of a given singularity. A method according to claim 7, characterized in that when a class is associated with a singularity after acquired skin images are provided to a neural network configured to deliver a classification of said images, at least one class is included among the following list of classes: ■ a class relating to the geometry of the periphery of the singularity of a given dermoscopic photo (IMDi, IMD2), ■ a class relating to the characterization of a geometry of the periphery of the singularity of a given dermoscopic photo (IMD1, IMD2) with respect to a plurality of characterizations of geometries of peripheries of singularities of other dermoscopic photos (IMD1, IMD2) considered in the vicinity of the given dermoscopic photo (IMD1, IMD2); ■ a class relating to the color of a singularity; ■ a class relating to the asymmetry of the geometry of the periphery of the singularity of a given dermoscopic image (IMD1, IMD2), ■ a class relating to the diameter of the geometry of the periphery of the singularity of a given dermoscopic image (IMD1, IMD2), when said singularity has a substantially circular shape, ■ a class relating to the area on the human body where the singularity is present. A method according to any one of claims 1 to 8, characterized in that an evolution criterion is calculated quantifying the evolution of a descriptor of a singularity between two images (IMi, IM2) from two acquisitions performed at two different dates (DATE-i, DATE2). Method according to claim 9, characterized in that an evolution criterion is calculated from a defined distance between a first value of a descriptor of a first node of a first graph acquired at a first date and a second value of a descriptor of a second node of a second graph acquired at a second date, each graph being generated from a first image, respectively from a second image, said images corresponding to a body of the same individual and the first and second nodes having the same position within the first and second images. Method according to claim 9, characterized in that the color and / or geometry of a symbol (Si, S2) is / are selected according to: ■ a criterion for a singularity to belong to at least one class of the classifier; ■ a value of a descriptor exceeding a threshold value; ■ the value of an evolution criterion for a singularity descriptor calculated between two initial images acquired at two dates (DATEo, DATE1). A method according to claim 1, characterized in that a third symbol (S3) is generated according to a given color and / or shape(s) when a singularity (SGi) is present in a first image acquired at a given position for the first time, said color or shape of the third symbol (S3) allowing said symbol to be distinguished from another symbol to indicate the new appearance of said singularity. A method according to any one of claims 1 to 12, characterized in that a user interaction with at least one displayed symbol (Si, S2, S3) allows the generation of a first digital instruction (CONS1) aimed at displaying at least one dermoscopic image (IMD1) in a display window, said dermoscopic image being displayed (IMDi) corresponding to an image extracted from the first image (IMi) associated with the position at which the symbol (Si, S2, S3) is displayed on the first image (IM1).

14. A method according to any one of claims 1 to 12, characterized in that a second digital instruction (CONS2) generated by a user action allows two dermoscopic images (IMD1, IMD2) to be displayed side by side, extracted respectively from a first image (IM1) and a second image (IM2), said two dermoscopic images (IMD1, IMD2) allowing the singularities of the same position on the body to be displayed according to the same resolution and according to the same scale of dimensions.

15. A method according to any one of claims 1 to 12, characterized in that a first digital command (CDN1) allowing movement, zoom, or selection of an area of ​​interest of the first image (IM1) of the first representation (PRES1) automatically causes the generation of an identical digital command (CDNo') on an equivalent area of ​​interest of the second image (IM2) of the second representation (PRES2).

16. A method according to any one of claims 1 to 15, characterized in that a second digital control (CDN2) allows a three-dimensional digital avatar (MODo) of an individual's body to be oriented so as to display a portion of the body, a third digital control (CDN3) allows said portion of the body displayed to be enlarged on an area of ​​interest, said area of ​​interest displaying a plurality of markers each having a position on the surface of the human body in a reference frame (Ro) associated with the digital avatar (MODo), each marker being associated with a singularity of the human body, a fourth digital control (CDN4) allows said marker to be selected to display a dermoscopic image extracted from the first image (IM1), said extracted image being defined around the position of the selected marker. A method according to any one of claims 1 to 16, characterized in that the dermoscopic images are acquired by an image-capture device configured to acquire a plurality of images of the skin of a human body and to assign each image a position on a 3D model (MODo) representing the body of said individual. A system comprising an electronic terminal including a display for generating the images produced by the method of any one of claims 1 to 17 and a data exchange interface for receiving images acquired by an image-capture device.

Images