Method for modelling an anatomical structure of the human body

Abstract

The invention relates to a method for modelling an anatomical structure of the human body in three dimensions that is computer-implemented and that comprises the following steps: - obtaining a plurality of consecutive images of the anatomical structure using a device for capturing images that is in movement and that bears a light source, - identifying anatomical sub-structures of the anatomical structure in the obtained images, - carrying out three-dimensional modelling of the anatomical structure based on the obtained images, - comparing the three-dimensional shape of the anatomical sub-structures of the obtained images with models of anatomical sub-structures stored in a database, step (6) of carrying out three-dimensional modelling of the anatomical structure based on the obtained images comprising predicting the depth of the anatomical structure and predicting the movement of the image-capturing device.

Description

[0001] Method of modeling an anatomical structure of the human body

[0002] The invention relates to methods and devices for imaging anatomical structures of the human body and more particularly to methods and devices enabling three-dimensional modeling of such structures.

[0003] The creation of three-dimensional models of anatomical structures for the diagnosis of various pathologies is increasingly being carried out in order to be able to diagnose these pathologies with precision. We will then take the example of the diagnosis of dry eye which can be caused by, among other things, smoking, wearing contact lenses, Parkinson's disease, lupus or even rheumatoid arthritis.

[0004] Physicians have low accuracy and reproducibility in quantifying dry eye, which complicates treatment assessment and prescription. Quantifying dry eye can involve a slit lamp examination. For example, the ophthalmologist may first apply a contrast agent to the cornea and then observe several signs of dryness. First, they measure the time it takes for the tear film to break after blinking, with a short time indicating a thin film. Then, they detect areas of the ocular surface damaged by dryness. These areas appear as dots, spots, or filaments: depending on their number and location, a degree of dry eye can be determined. Dyes can also be used to highlight ocular lesions of the sclera and cornea.The problem with these measurements is that they are imprecise and poorly reproducible. These limitations prevent reliable quantification of the evolution of dry eye in a patient. In particular, they do not allow the effect of a treatment to be measured satisfactorily.

[0005] There are also other diagnostic methods that do away with the slit lamp examination to diagnose dry eye: dedicated devices based on different imaging principles, such as meibography, are being developed. The slit lamp, which every ophthalmologist is equipped with, must then be replaced with a new device.

[0006] In all cases (for an eye examination or any other type of examination), a video is acquired (by slit lamp, endoscope or even by a surgical microscope). Principles of egomotion (related to the three-dimensional movement of a camera relative to its environment) are implemented for the creation of three-dimensional models from videos acquired by an imaging device such as those mentioned above. However, these methods pose a problem linked to the fact that the implemented processes violate a founding principle of structure determination techniques based on movement (or "Structure from Motion"), namely that two corresponding structures in two consecutive images of a video will have the same color and the same intensity.However, in the above-mentioned images, the lighting is related to the camera, not the scene, which invalidates the hypothesis (the results of the above-mentioned methods are highly dependent on the light and shadows of an analyzed scene). In addition, modeling errors may appear depending on the modeling cases.

[0007] The invention aims in particular to provide a method for modeling an anatomical structure of the human body making it possible to produce three-dimensional models that are more reliable and more repeatable than the solutions proposed in the prior art.

[0008] To this end, the invention relates to a method for modeling an anatomical structure of the human body in three dimensions implemented by computer and comprising the following steps:

[0009] - Obtaining several consecutive images of the anatomical structure using a moving image capture device carrying a light source, the light source being configured to illuminate the anatomical structure,

[0010] - Identification of anatomical substructures of the anatomical structure in the obtained images,

[0011] - Creation of a three-dimensional model of the anatomical structure from the images obtained,

[0012] - Comparison of the three-dimensional shape of the anatomical substructures of the obtained images with models of anatomical substructures stored in a database, the step of carrying out a three-dimensional modeling of the anatomical structure from the obtained images including a prediction of the depth of the anatomical structure as well as a prediction of the displacement of the image-taking device.

[0013] Thus, we obtain a three-dimensional "Structure from Motion" type modeling process (movement of the image capture device) of anatomical structures of the human body, firstly comprising a step of semantic segmentation of the captured images into substructures of interest with verification of the consistency of these substructures with substructures of a database and which represent, in terms of shape, the type of substructures to be observed. This makes it possible to correct any modeling error by comparison with known and proven structures. These techniques, called "3-D shape matching" (or correspondence of shapes in three dimensions), make it possible to produce reliable and repeatable modeling.

[0014] In short, we use prior knowledge of the substructures to be observed at a structure of interest and an assumption derived from the fact that the light source is carried by the imaging device to reliably and repeatably model a three-dimensional anatomical structure. This results in an image with very good resolution, a wide field of view and without illumination artifacts.

[0015] According to other optional characteristics of the modeling method taken alone or in combination: the step of obtaining several consecutive images of the anatomical structure comprises the recording of a video followed by a cutting of this video into several consecutive images; the modeling method is a self-supervised method; the modeling method comprises a step of feeding an artificial intelligence with models of substructures stored in a database; the modeling is carried out in real time; the step of carrying out a three-dimensional modeling of the anatomical structure from the images is a modeling of the identified anatomical substructures;the step of comparing the three-dimensional shape of the anatomical substructures of the obtained images with models of anatomical substructures stored on a database comprises the use of the following loss functions: semantic reconstruction loss, sphere fitting loss, shape preservation loss, and preferably photometric loss, structure similarity loss as well as depth smoothing; the modeling method is a three-dimensional modeling method of an eye; and the anatomical substructures are the cornea and the sclera.;

[0016] The invention also relates to a computer program comprising instructions which, when the program is executed by a computer, cause the latter to implement the steps of the method according to the invention.

[0017] The invention also relates to a computer-readable recording medium comprising instructions which, when executed by a computer, cause the latter to implement the steps of the method according to the invention.

[0018] The invention also relates to a method for detecting a pathology resulting from the observation of a structure of the human body comprising:

[0019] A modeling method according to the invention, and

[0020] A step in diagnosing a pathology based on the modeling carried out.

[0021] The invention also relates to a device for modeling an anatomical structure of the human body in three dimensions for implementing a method according to the invention and comprising:

[0022] - a device for capturing moving images and carrying a light source, - means for producing a three-dimensional model of the anatomical structure from the images obtained,

[0023] - means of identifying anatomical substructures of the anatomical structure in the obtained images, and

[0024] - means of comparing the three-dimensional shape of the anatomical substructures of the obtained images with the substructure models stored in a database.

[0025] Preferably, the image capturing device is formed by a camera configured to record a video.

[0026] Brief description of the figures

[0027] The invention will be better understood on reading the following description, given solely by way of example and with reference to the appended drawings in which:

[0028] Figure 1 is a schematic representation of the method according to the invention, and

[0029] Figure 2 is a representation of a model according to the invention.

[0030] Detailed description

[0031] We will now describe a modeling method according to the invention step by step. In the following description, the example of a three-dimensional modeling of an eye will serve as an example to illustrate the different steps. As explained above, the method and the modeling device can be implemented for modeling other anatomical structures of the human body. We will also describe in this section a self-supervised method, advantageous in our case. It is recalled here that a self-supervised method is a machine learning method based on a neural network designed to ingest unlabeled data.

[0032] The method described here is preferably carried out in real time.

[0033] The first step 2 consists of obtaining several consecutive images of the anatomical structure using a moving image-taking device carrying a light source configured to illuminate the anatomical structure. The image-taking device may be carried by a slit lamp, an endoscope, etc. The light source may be of any known type allowing the anatomical structures to be modeled to be illuminated. These known means will not be detailed in this application. The movement of the camera involves the implementation of a process called “Structure from Motion” as explained above.

[0034] The imaging device can be a video camera. It could also be a device that takes snapshots as an alternative to a video. For example, in the case of an eye examination, videos can be recorded using a Haag Streit BQ 900 slit lamp and the CM 900 camera module (resolution 1600*1200, 12 frames per second). The magnification can be set to *10 and the ocular surface analyzed after illumination with white light, followed by cobalt blue light and the interposition of a yellow filter.

[0035] In the case of a video, a step of cutting the video into several images is carried out. This could, for example, be sequences of three images with an interval of ten images between each sequence (this could, for example, be the case for a camera at 12 images per second).

[0036] The selected images are then processed to predict the semantic segmentation of each of them (step 4). To do this, a segmentation network trained on the basis of a "Feature Pyramid Network" type model can be used (used for object detection) to predict this semantic segmentation of the anatomical structure to be modeled into substructures of interest, by labeling the pixels belonging to one anatomical substructure or another. These can for example be, in the case of the eye, the cornea and the sclera. Other structures can be deliberately not retained because they are a source of modeling problems. To do this, each pixel in the following classes can be classified: eyelid, sclera, cornea.All pixels labeled "eyelid" are ignored in training and inference, as they do not contain any useful information for modeling (the eyelid also moves relative to the eyeball, which goes against the assumption underlying the described model and cannot be modeled by a rigid (spherical) shape model).

[0037] The next step 6 is to perform a three-dimensional modeling of the anatomical structure from the images. This prediction can be achieved by using two convolutional neural networks (in the case of a self-supervised process) that can be used to predict depth (DepthNet convolutional neural network) and the displacement of the image-taking device, or egomotion (EgomotionNet convolutional neural network). The DepthNet and EgomotionNet architectures used are those proposed in the paper “Casser, V., Pirk, S., Mahjourian, R., & Angelova, A. (2019). Unsupervised monocular depth and ego-motion learning with structure and semantics. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition Workshops (pp. 0-0)”.

[0038] The DepthNet convolutional neural network uses a fully convolutional encoder-decoder architecture to obtain a dense depth map from a single Red-Green-Blue image. It can be represented by the following equation: in which the i is the image at time i, D i is the depth map associated with the image, H is the height of the image, and W is the width of the image.

[0039] The convolutional neural network EgomotionNet, for example, takes three consecutive images and predicts transformations simultaneously. Both neural networks are used for inference and artificial intelligence training in a self-supervised process. This modeling makes it possible to visualize possible damaged areas on the surface of the eye. It can be represented by the following equations: in which the i is the image at time i, l i+nis the image at time i+n, , t is the translation in the x, y, z directions, and r is the rotation in the x, y, z directions.

[0040] Several loss functions are considered for the realization of an accurate and reproducible three-dimensional modeling. These allow the implementation of step 8 of comparison of the three-dimensional shape of the anatomical substructures of the obtained images with substructure models stored in a database.

[0041] The first and main one is the semantic reconstruction loss (SRL) which is the main supervisory signal. It can be represented by the following equation: in which Sj is the segmentation of image j, and Si → j is the segmentation of image i aligned with the segmentation of image j.

[0042] The second is photometric loss (or "photometric loss", recon), is similar to semantic reconstruction loss except that images are used as input points to compare a reconstructed image with the next image. It can be represented by the following equation: in which the j is the image j, and l i → j is image i aligned with image j.

[0043] The third is structural similarity loss (SSIM), which is implemented to compare the luminance, contrast, and structure between a reconstructed image and the next image. It can be represented by the following equations: in which μ i , μ j are the mean, σ 2 i, σ 2 j the variance, and σ ij the covariance of x,y, L is the dynamic range of pixel values, k1= 0.01, and k2= 0.03.

[0044] The fourth is depth smoothness (DS) which encourages smoothing by penalizing depth discontinuity if the image has continuity in the same area.

[0045] It can be represented by the following equations: in which the i is the image i, D i is the depth map associated with image l i , are the gradients of the image l i in the horizontal and vertical direction.

[0046] One of the main ones, forming in particular the step of comparing the three-dimensional shape of the anatomical substructures of the images obtained with substructure models stored in a database, is the sphere fitting loss (or "sphere fitting loss"). In the case of an eye examination, the latter is implemented for what is estimated to be the shape of the eye: two intersecting spheres. A smaller anterior sphere represents the cornea and a posterior sphere represents the sclera. In order to implement this loss, we first estimate a depth map, then using semantic segmentation it is possible to estimate the sphericity of the two regions: cornea and sclera. We can also apply a threshold on the surface before calculating this loss to ensure that both anatomical substructures are present.Once an image meets the threshold, it is possible to use the depth estimates for the cornea or sclera to convert this into a three-dimensional point cloud projection using the inverse of the intrinsic matrix. It is then possible to use an estimated point cloud and apply a least-squares sphere fit. It is then possible to determine the best sphere center for the given data points. By rearranging the terms of the following equation:. in which x,y,z are points in space, x0, y0, z0 are the coordinates of the center of the sphere, and r is the radius of the sphere, it is possible to express the equation in matrix notation and solve the following equation for the term:

[0047] In which: in which X k ,y k ,Z kis one of the data points, x0, y0, z0 are the coordinates of the center of the sphere, r is the radius of the sphere.

[0048] By fitting the n data points X k , y k , Z k , it is possible to solve for the coordinates of the center of the sphere x0, y0, z0 and the radius r.

[0049] Sphere fitting loss is a root mean square error between the fitted sphere and the data points. The sphericity for each of the corneal and scleral regions has an α weight e . With both regions fitted to a sphere, it is then possible to calculate the loss for each pixel p. The corresponding equations are: in which pc is the number of pixels on the corneal surface and ps the number on the scleral surface, xck the data points on the cornea, xcO the coordinates of the center of the cornea, xsk the data points on the sclera, xsO the coordinates of the center of the sphere of the sclera.

[0050] The above set of loss functions are used to obtain a model using a differentiable image warping operator taking into account a Red-Green-Blue image îi, a depth mapping D j , and an egomotion Ei->j, allowing to obtain a reconstructed image îjth. This is modeled by the following equation:

[0051] A fifth is structural similarity loss (SSIM), £ scl, implemented to take into account the temporal context of videos, namely to track and ensure that the intrinsic attributes of the scene remain unchanged over time. In the case of eye examinations, it models the fact that the shape of a patient's ocular surface, and in particular the radius of the two spheres, remains unchanged during an examination. The objective is to make shape predictions more robust, less sensitive to variations in illumination, eyelid opening, etc. In this case, we randomly select N images within the same video, at times i1, i2, ... , I N . Then, we consider the predicted radius for the cornea and the sclera, for each of these images: r c1 , r c2 , .... r c N and r s1 , r s2 , ... , r sN, respectively. The objective is to minimize the standard deviation of the radius predictions of the spheres within each of the two sets. The corresponding equations are:

[0052] Taking into account the shape-preserving loss function added together with all the other loss functions then used in the above equation, the equation is modeled as follows:

[0053] The database mentioned above can be included in local or remote storage means. The objective of the step of comparing the three-dimensional shape of the anatomical substructures of the images obtained with substructure models stored on a database is to verify that the three-dimensional modeling carried out above is consistent, at the level of the general shapes of the substructures of interest, with a physical reality corresponding to the models stored on the database. In the case of a self-supervised process, it is a question of feeding an artificial intelligence with known and robust data with training

[0054] Example of implementation After taking images and semantic segmentation using the devices (video camera) and networks (“Feature Pyramid Network”) described above, the data is processed using the convolutional neural networks described above and the different loss functions in order to produce a three-dimensional model, here of a human eye.

[0055] A first step in this example consisted of manually annotating points corresponding to damaged areas on the surface of the eye, and the visible veins of the sclera. To visualize the accuracy of the predictions made (semantic segmentation, depth, egomotion), a source image was transformed into a target image and the marked points were tracked. Using the equation in which p s corresponds to the pixels of the source image, p t corresponds to the pixels of the target image (target), D t correspond à ladepth map of the target image, T t -> s is the transformation matrix, K the intrinsic matrix, and K -1 the inverse of the intrinsic matrix. p pixels s of the source frame are projected onto the target frame p t The evaluation performed used inverse deformation which first requires the prediction of the depth map D t of the target image, and then the egomotion from source to target which also gives the transformation matrix T t -> s The evaluation metric was implemented on a test set of 3 patients with 54 images. The test set consists of 126 points on the sclera on vein intersections and punctuation points, 33 points on the cornea of ​​visible punctuation points.

[0056] The baseline (or reference method according to the prior art) used for comparison had to be trained intermittently due to the infinitely divergent loss, as well as the irreversibility of the predicted egomotion matrix.

[0057] The baseline was trained using a learning rate of 0.0002, a batch size of 8, an SSIM weight of 0.15, an L1 reconstruction weight of 0.85, and a depth smoothing weight of 0.04. The training setup according to the invention uses a learning rate of 0.0002, a batch size of 8, an SSIM weight of 0.15, an L1 reconstruction weight of 0.15, an L2 semantic reconstruction weight of 0.04, a sphere fitting weight of 10000, and a depth smoothing weight of 0.04 [3],

[0058] All models were trained for up to 200 epochs (a hyperparameter of gradient descent that controls the number of complete passes through the training dataset), with the best model (lowest loss achieved) being kept for inference. Table 1 below shows the results of evaluating our test set by different methods. By defining two of the new loss functions (sphere fitting and semantic segmentation), it is possible to stabilize the training and prevent the losses from diverging. Better results are also obtained as shown below. Despite the increased computation during training via the loss functions, the inference time remains similar to that of the baseline. In the following experiments, we preprocessed the data using a frame size of n = 10 (including the baseline experiment).

[0059] We compared three different configurations according to the invention:

[0060] “SiGMoid n°3” in which only semantic segmentation is used,

[0061] “SiGMoid No. 4” in which semantic segmentation and sphere fitting were used, and

[0062] “SiGMoid #5” with another convolutional neural network used.

[0063] The table above represents the results obtained using five methods: the three described above and two methods according to the prior art. The average Euclidean distance in pixels is the lowest in SiGMoid No. 4. The method according to the invention implementing semantic segmentation, the convolutional neural networks described above as well as the two added loss functions achieves the lowest error compared to the baseline. This more precise and more robust modeling therefore allows a better diagnosis of a pathology by a doctor.

[0064] Figure 2 illustrates an image 11 of a human eye, a representation of a semantic segmentation 11' of this eye into two anatomical substructures which are the cornea 12 and the sclera 14, and a modeling 16 of the eye using the “SiGMoid n°4” method.

[0065] The device for modeling an anatomical structure of the human body in three dimensions includes, as listed above:

[0066] A device for capturing moving images and carrying a light source. A non-limiting example (and used for the production of the example above) has been detailed above; Means for producing a three-dimensional model of the anatomical structure from the images obtained;

[0067] Means for identifying anatomical substructures of the anatomical structure in the obtained images; and

[0068] Means for comparing the three-dimensional shape of anatomical substructures of the obtained images with the substructure models stored in a database.

[0069] The various means enabling the above modeling, identifications and comparisons may be formed by an electronic control unit comprising a non-volatile memory storing the various predetermined control programs and potentially the database comprising robust models. These means include, among others, conventional computer means, or conventional computer computing means, such as processors, RAM and hard memories. These means execute the instructions of computer programs, i.e. executable codes, recorded on computer-readable media.

[0070] The invention is not limited to the embodiments presented and other embodiments will become clear to those skilled in the art. As explained above, the modeling method can be implemented to model anatomical structures other than the eye. In addition, the technical means described for taking images or processing information may differ from those described.

[0071] List of references

[0072] 2: step of obtaining several consecutive images of the anatomical structure using a moving image capture device carrying a light source

[0073] 4: step of identification of anatomical substructures of the anatomical structure in the obtained images

[0074] 6: step of creating a three-dimensional model of the anatomical structure from the images obtained

[0075] 8: step of comparing the three-dimensional shape of the anatomical substructures of the obtained images with substructure models stored in a database

[0076] 11: image

[0077] 11': semantic segmentation

[0078] 12: cornea

[0079] 14: sclera

[0080] 16: modeling

Claims

Demands

1. A method for modeling a three-dimensional anatomical structure of the human body implemented by computer and characterized in that it comprises the following steps: - Obtaining (2) several consecutive images of the anatomical structure using a moving image-capturing device carrying a light source, the light source being configured to illuminate the anatomical structure, - Identification (4) of anatomical substructures of the anatomical structure in the images obtained, - Creation (6) of a three-dimensional model of the anatomical structure from the images obtained, - Comparison (8) of the three-dimensional shape of the anatomical substructures of the images obtained with models of anatomical substructures stored on a database, the step of realization (6) of a three-dimensional modeling of the anatomical structure from the images obtained including a prediction of the depth of the anatomical structure as well as a prediction of the displacement of the image-taking device.

2. Modeling method according to claim 1, wherein the step of obtaining (2) several consecutive images of the anatomical structure comprises recording a video followed by cutting this video into several consecutive images.

3. A modeling method according to any one of the preceding claims, the modeling method being a self-supervised method.

4. A modeling method according to the preceding claim, comprising a step of feeding an artificial intelligence with models of substructures stored on a database.

5. A modeling method according to any one of the preceding claims, the modeling being carried out in real time.

6. A modeling method according to any one of the preceding claims, wherein the step of realizing (6) a three-dimensional model of the anatomical structure from the images is a modeling of the identified anatomical substructures.

7. A modeling method according to any one of the preceding claims, wherein the step of comparing (8) the three-dimensional shape of the anatomical substructures of the images obtained with models of anatomical substructures stored in a database includes the use of the following loss functions: semantic reconstruction loss, sphere fitting loss, shape preservation loss, and preferably photometric loss, structure similarity loss, and depth smoothing.

8. A modeling method according to any one of the preceding claims, the method being a three-dimensional modeling method of an eye.

9. A modeling method according to the preceding claim, wherein the anatomical substructures are the cornea and the sclera.

10. A computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the steps of the process according to any one of the preceding claims.

11. A computer-readable recording medium comprising instructions which, when executed by a computer, cause the computer to carry out the steps of the process according to any one of claims 1 to 9.

12. A method for detecting a pathology resulting from the observation of a structure of the human body comprising: - A modeling method according to any one of claims 1 to 9, and - A diagnostic step for a pathology based on the modeling performed.

13. A device for modeling a three-dimensional anatomical structure of the human body for implementing a method according to any one of claims 1 to 9, characterized in that it comprises - A device for capturing moving images and carrying a light source, - Methods for creating a three-dimensional model of the anatomical structure from the images obtained, - Means of identifying anatomical substructures of the anatomical structure in the images obtained, and - Means of comparing the three-dimensional shape of the anatomical substructures of the images obtained with the models of substructures stored in a database.

14. Modeling device according to the preceding claim, wherein the image-capturing device is formed by a camera configured to record a video

Images