Method for acquiring data by endoscopy from biological tissues, and corresponding endoscopic imaging system.

Abstract

The invention relates to a method for analyzing images acquired by endoscopy from biological tissues using a light-generating device associated with an endoscope with a camera. [Fig.4]

Description

Title of the invention: Method for acquiring data by endoscopy from biological tissues, and corresponding endoscopic imaging system. Field of invention

[0001] The field of the invention is that of medical imaging.

[0002] More specifically, the invention relates to the acquisition of data from imaging resulting from the endoscopic exploration of biological tissues, and more specifically, of hollow organs or cavities of humans or animals. Prior art

[0003] In the medical field, endoscopes are of crucial importance in enabling visual observation of hollow organs and cavities of the human body, and remain essential for patient monitoring, diagnosis and interventional procedures.

[0004] Endoscopes are particularly useful for detecting lesions or signs of inflammation present on the surface of tissues. In fact, various cancers of hollow organs are the consequence of a carcinological process resulting from chronic inflammation. The pre-cancerous lesions at the origin of these cancers are difficult to detect due to their appearance, which is not very different from healthy mucosa, and their flat nature. Thus, very often, these lesions are not detected and the diagnosis is made at the cancer stage, which leads to heavy care and a reduction in the survival rate of patients.

[0005] In recent years, medical imaging techniques have been developed to attempt to detect tissue changes early, and in particular pre-cancerous or cancerous lesions.

[0006] There are methods of illuminating the surface of the organ to be studied using a light source, while moving the endoscope in the human cavity, and acquiring a video by a color camera. Such a method makes it possible to obtain a video reflecting the reflectance of the light by the illuminated tissue.

[0007] The color camera records the light signal in the form of three bands (red, green, and blue) covering the visible spectrum, while the light source is either white light or light composed of two or four wavelength bands.

[0008] The visuals obtained differ depending on the light source used. Video acquisition under white light results in images similar to what the human visual system can perceive. On the contrary, video acquisition under a composite light of two or four wavelength bands, allows false color images to be obtained from a combination of the images.

[0009] However, current systems provide true or false color images whose textures, particularly of the mucous membranes, are insufficiently defined and recognizable, which increases the risk for the operator of not detecting lesions. In addition, such systems do not provide any assistance in interpreting the images.

[0010] There is therefore an unmet need to provide a method and device for acquiring endoscopic imaging making it possible to obtain images of improved quality and to provide data which can subsequently be exploited, in particular by artificial intelligence. Summary of the invention

[0011] The present invention makes it possible to solve the problems raised by the prior art.

[0012] For this purpose, the present invention proposes a method for acquiring data by endoscopy from biological tissues using a light generating device associated with an endoscope with camera, said method comprising: - the generation of light on an area of ​​biological tissue by a light device capable of generating n successive spectral bands of different wavelengths so as to cover the visible spectrum and the near infrared and generate a multispectral video in which each image corresponds to one of the spectral bands; - cutting the endoscopic video thus obtained so as to extract images from spectral bands of different wavelengths; - the registration of the spectral band images to obtain at least one registered multispectral image comprising an image for each spectral band of wavelengths; - the spectral reconstruction of a reflectance cube, from the recalibrated multispectral image, in which each pixel corresponds to a vector reflecting the reflectance of the biological tissue area.

[0013] The images obtained in the spectral bands covering the near infrared are advantageously used to characterize the tissues in a more absolute manner by complementarity with the spectral bands covering the visible spectrum.

[0014] This multiplicity of spectral bands covering the visible spectrum and the near infrared allows, in a first step, the obtaining of a multispectral image having an improved and extended spectral resolution with respect to color images, and in a second step, to generate a reflectance cube in which each pixel corresponds to a vector reflecting only the reflectance of the tissue zone. corresponding. Compared to images acquired by the n spectral bands, such reflectance cubes also include better spectral resolution over the visible and near infrared range in each pixel specific to the reflectance of the tissue after having freed itself from the radiance of the illuminant and the sensitivity of the camera.

[0015] Advantageously, the wavelength bands may be between 380 and 1000 nm, preferably between 400 and 850 nm.

[0016] Advantageously, n may be equal to six or a multiple of six.

[0017] Advantageously, at least four of the spectral bands may have wavelengths covering the visible spectrum and at least one of said spectral bands has wavelengths covering the near infrared.

[0018] According to a particular characteristic, at least four of said spectral bands may have wavelengths covering the visible spectrum and at least two of the spectral bands may have wavelengths covering the near infrared.

[0019] The use of a second spectral band covering the infrared makes it possible to increase the quality of the multispectral image obtained.

[0020] Advantageously, the camera can capture 20 to 30 images per second during the acquisition of the multispectral video, and only one image out of two can be alternately selected for the production of a multispectral image during the cutting of the multispectral video obtained.

[0021] According to a first advantageous embodiment of the method according to the invention, the spectral reconstruction of a reflectance cube from the multispectral image can comprise processing of the multispectral image by a deep neural network.

[0022] According to a second advantageous embodiment of the method according to the invention, the spectral reconstruction of a reflectance cube from the recalibrated multispectral images can comprise: - a second registration (or “mosaicing”) of the registered multispectral images in order to obtain a mosaic of multispectral images; - processing of the multispectral image mosaic by a deep neural network or by a light-tissue interaction model.

[0023] Mosaicking a set of images consists of positioning the different images in a common reference frame then covering a field of view larger than that of the images taken individually.

[0024] The implementation of the method according to the second embodiment makes it possible to obtain several multispectral images representing a common area of ​​the observed biological tissue. Thus, the generation of a reflectance spectrum can be carried out from several reflectance data vectors of the same area but acquired with a different positioning of the endoscope, which allows to improve the estimation of the reflectance of the biological tissue.

[0025] Preferably, whether for the first or the second embodiment, the deep neural network can be pre-trained from reflectance spectra data characteristic of healthy and pathological tissues.

[0026] The method may further comprise a histological analysis associated with the area of ​​the tissue, in particular by microscopic analysis. This histological analysis provides ground truth potentially necessary for learning for the purposes of automatic analysis or characterization of tissues.

[0027] Advantageously, the method may further comprise analyzing the reflectance cube by artificial intelligence.

[0028] It should be noted that the method according to the present description can be implemented indifferently in real time, that is to say during the endoscopic examination, or in a deferred time, that is to say after the endoscopic examination.

[0029] The present invention also relates to an endoscopic imaging system which comprises: - an endoscope configured to capture a video stream, - at least one box generating light from an LED source, and - a display screen configured to display images acquired by the endoscope and / or all or part of the information contained in the reflectance cubes obtained with a data acquisition method as described previously. List of figures

[0030] Other characteristics and advantages of the invention will appear more clearly on reading the following description of preferred embodiments, given as illustrative and non-limiting examples, and the appended figures among which: - [Fig.l] illustrates the main steps of a method of acquiring data by endoscopy from biological tissues according to the invention; - [Fig.2] illustrates an example of non-rigid interband registration according to the present description; - [Fig.3] illustrates the main steps of the method of [Fig.l], according to a first embodiment; - [Fig.4] illustrates the main steps of the method of [Fig.l], according to a second embodiment; - [Fig.5] illustrates the main steps of the process of [Fig.4], according to a variant; and - [Fig.6] illustrates the absorption spectra of hemoglobin. Detailed description of the invention

[0031] The general principle of the method of acquiring data by endoscopy from biological tissues is described in more detail in relation to [Fig.l].

[0032] During a step 11, a light device associated with an endoscope generates light with n spectral bands of different and successive wavelengths so as to generate a multispectral video in which each image corresponds to one of the spectral bands.

[0033] More precisely, this step 11 comprises the generation of a light 111 having wavelength bands covering the visible spectrum and the near infrared, that is to say so as to respectively cover wavelength ranges between 380-780 nanometers (nm) and 780-1000 nm.

[0034] The light device can illuminate the tissue to be observed successively with the different spectral bands, or alternatively, can illuminate the tissue to be observed with a plurality of spectral bands.

[0035] Each of the spectral bands can cover a wavelength range from a few nanometers to several tens of nanometers, for example a wavelength range of 50 nanometers, preferably a wavelength range of 40 nanometers.

[0036] The spectral bands of the generated light are preferably six in number, or a multiple of six. Thus, it is possible to select at least four bands covering the visible spectrum and at least one spectral band covering the infrared.

[0037] The four spectral bands covering the visible spectrum make it possible to obtain images each comprising different absorption peaks.

[0038] The precision of these results is refined by the images obtained with the infrared spectral band(s). Indeed, the spectral bands covering the infrared make it possible to obtain additional absorption peaks, and thus offer the possibility of obtaining information that cannot be perceived with spectral bands covering the visible spectrum.

[0039] By way of example, the following six wavelength bands can be used: a first band centered on 415 nm, a second band centered on 450 nm, a third band centered on 540 nm, a fourth band centered on 605 nm, a fifth band between 650 and 700 nm, a sixth band between 700 and 750 nm.

[0040] According to a first alternative, the light emitted by the light device replaces the light emitted by the endoscope, the latter being consequently switched off during the implementation of the present method. In other words, in this alternative, the white light can be switched on and used to position the camera of the endoscope. in the organ and area to be observed, and turned off during video recording.

[0041] According to a second alternative, the light emitted by the light device complements the light emitted by the endoscope. In other words, the white light of the endoscope remains on during the acquisition of the video.

[0042] This second alternative can in particular be implemented when the power of the LED source is sufficiently high to illuminate the observed tissue area and acquire images characteristic of the spectral bands. Indeed, the LED source must guarantee that the relevant details of the spectral band of a given wavelength range are captured despite the presence of the white light of the endoscope.

[0043] Step 11 further comprises the acquisition of a multispectral video 112 by the camera of the endoscope, as well as successive illumination 113 of the area observed by the different spectral bands. This results in an endoscopic video in which each image corresponds to one of the spectral bands with which the area of ​​tissue was illuminated.

[0044] This successive lighting is carried out in a synchronized manner with the camera rate. To do this, endoscopic camera rate information from the video stream is recovered so as to generate a timer based on an electronic device which allows the triggering of the spectral bands emitted by the light source.

[0045] During a step 12, a cutting 121 of the endoscopic video obtained is carried out so as to extract images of spectral bands of different wavelengths,

[0046] The video in fact needs to be cut in such a way as to eliminate the so-called “transition” images. “Transition” images are the images obtained between two successive illuminations by different spectral bands, and which generally present unusable data resulting from the change of spectral bands.

[0047] The camera can, for example, capture 20 to 30 images per second during video acquisition. When the camera has this latter rate, a cutting step can be carried out so as to select one image out of two of the endoscopic video. This cutting results in one or more multispectral images. Each multispectral image comprises at least one image representative of each of the spectral bands.

[0048] The size of the images can be chosen so that the images can encompass one or more lesions present on the surface of the tissue. The dimensions of the images can vary depending on different parameters such as the acquisition conditions, the nature of the organ studied and the pathological areas likely to develop there.

[0049] During a step 13, the images resulting from the cutting undergo a recalibration 132 so as to obtain one or more multispectral image(s) comprising an image for each spectral band of wavelengths.

[0050] The images of the spectral bands of the same multispectral image are superimposed on each other so that the characteristic elements of the analyzed tissues, present in each of them, are aligned. The characteristic elements are all the variations of tissues which may be on the surface of the observed organs, and may be for example lesions, signs of inflammation, blood vessels or variations in tissue granulosity.

[0051] In particular, a so-called “non-rigid” interband registration is carried out. In the present description, “non-rigid interband registration” means an elastic and anisotropic registration for all the pixels to respond to the elastic deformations of the observed tissues.

[0052] Thus, unlike rigid interband registration which consists of aligning all the characteristic elements of an image by a single geometric transformation composed of a rotation and a translation, non-rigid interband registration consists of aligning each characteristic element obtained in the different spectral bands independently in one or more of the images of the multispectral image. Indeed, the movements of the observed organ coupled with those of the endoscope, in particular when it is a stomach, are likely to cause a displacement in space of the observed characteristic element, and thus, the characteristic element has a different position and shape in each image of the multispectral image.

[0053] By way of illustration, [Fig.2] represents an example of non-rigid interband registration of a point A in a multispectral image. [Fig.2] shows that before registration, one of the points A (corresponding to a characteristic element) is not aligned with the other corresponding points A, despite the alignment of the different images constituting the multispectral image. This type of shift in the positioning of point A may in particular be the result of a movement of the internal wall of the organ observed. In order to take this movement into account, the non-aligned point A is recentered so as to be aligned with the corresponding points A.

[0054] This type of registration can be carried out by a geometric approach. The latter is based on the extraction from each of the images of geometric primitives which are paired in order to determine the transformation between the images two by two, for example in the form of a transformation matrix. It can also be carried out by a calculation providing an optical flow between the images two by two in order to determine a displacement vector field at each pixel.

[0055] This step results in a realigned multispectral image in which each pixel corresponds to a vector with n components and preferably corresponding to the number of spectral bands used. Each of the components then corresponds to one of the spectral bands.

[0056] Thus, each pixel of a multispectral image has n reflectance values ​​for the same point of the observed tissue. However, although these values ​​partly integrate the reflectance of the observed tissue, they also depend on the characteristics of the lighting device used and the sensitivity of the endoscope camera.

[0057] During a step 13, a spectral reconstruction of a reflectance cube from the obtained realigned multispectral image is implemented. More precisely, a reflectance cube is obtained in which each pixel corresponds to a vector reflecting only the reflectance of an area of ​​the observed tissue or organ.

[0058] In the present description, the reflectance is defined by the quantity of energy re-emitted relative to the quantity of energy received, and this for each of the wavelengths covered by the spectral bands.

[0059] This vector corresponds to the reflectance spectrum of the tissue for the pixel concerned over a given wide wavelength range, i.e. between 380-780 nm and 780-1000 nm. Each of its values ​​corresponds to the reflectance for a narrow band of the wavelength range which can be from a few nanometers to several tens of nanometers.

[0060] This spectral reconstruction can be carried out from one or more multispectral images as described previously, according to one of the embodiments presented below.

[0061] According to a first embodiment of the method for acquiring data by endoscopy from biological tissues represented by [Fig.3], the spectral reconstruction comprises processing of the multispectral image by a deep neural network.

[0062] The deep neural network is trained from a large amount of data so that the neural network generates reflectance spectra from multispectral images. The amount of data is preferably at least a few hundred.

[0063] Training the deep neural network involves providing it with a set of data pairs. Each pair is composed of a vector similar to those present in the multispectral images (input) and a vector similar to those present in the reflectance cubes (output) and representing the theoretical reflectance spectrum corresponding to the input data.

[0064] Alternatively, or in addition, the deep neural network can be pre-trained from pairs of data from healthy tissues and tissues pathological or abnormal, whose reflectance spectrum comes from an additional device.

[0065] According to a second embodiment of the method for acquiring data by endoscopy from biological tissues represented by [Fig.4], the spectral reconstruction comprises a registration of the multispectral images so as to obtain a corresponding mosaic of multispectral images, and a processing of the mosaic by a deep neural network.

[0066] The spectral reconstruction in this second embodiment requires the use of a plurality of multispectral images, which can be obtained successively over time. Preferably, the images forming the different multispectral images are obtained by the same spectral bands.

[0067] The multispectral image mosaic is processed by a new neural network. The neural network has been previously trained to provide a reflectance cube (output) from a multispectral image mosaic (input).

[0068] As described previously, the deep neural network can be previously trained from pairs of data from healthy tissues and pathological or abnormal tissues, the reflectance spectrum of which comes from an additional device.

[0069] [Fig.5] represents a variant of the second embodiment of the method previously presented. According to this variant, spectral reconstruction includes the registration of multispectral images so as to obtain a corresponding multispectral image mosaic, and processing of the mosaic by a light-tissue interaction model.

[0070] The light-tissue interaction model is an analytical model described by equations. It is obtained after characterization of each of the tissues from their optical properties such as light absorption and scattering.

[0071] In particular, the equations of the light-tissue interaction model include characteristic values ​​for each tissue.

[0072] For example, it is possible to determine with certain spectral bands of specific wavelengths the quantity of hemoglobin present in an area of ​​the observed tissue. This information concerning the quantity of hemoglobin can give an indication of the presence of an anomaly, a cancerous or precancerous lesion.

[0073] The propagation of light in a layer of tissue can be modeled via two optical processes: diffusion (defined by its reduced diffusion coefficient p's) and absorption (defined by its coefficient pa).

[0074] With this model, the reflectance R can be described according to the following formula:

[0075] [Math.l] R-R0 (jta. fi's)

[0076] RO is based on light transport theory (scattering theory or Monte Carlo simulations). The absorption spectra of hemoglobin are shown in [Fig.6].

[0077] Preferably, the spectral reconstruction according to the first or second embodiment results in a reflectance cube of spatial size identical to the acquired images, but each pixel of which corresponds to a spectral vector of dimension greater than the number of initial bands.

[0078] It is also possible to obtain a reflectance cube of spatial size smaller than the acquired images. Indeed, the captured images may represent the same area with a more or less significant shift, and consequently one or more parts of an image may be absent from the other images intended to form the multispectral image. The registration of the images between them may thus require reducing the sizes of the images so that they have identical sizes between them and represent the same and unique area of ​​biological tissue.

[0079] On the contrary, when several multispectral images are acquired, the spatial size of the reflectance cube can be greater than the size of the acquired images. The multispectral images constituting the mosaic comprise a plurality of images covering the observed tissue area, which makes it possible to construct a reflectance cube whose size results from the combination of the different multispectral images.

[0080] According to another aspect, an endoscopic imaging system intended to implement the method according to the present description comprises an endoscope, one or more boxes generating light from an LED source, and a display screen.

[0081] In particular, the endoscopic video may be acquired using an endoscopic imaging system, comprising an endoscope and a viewing screen.

[0082] The endoscope conventionally comprises an operating channel provided with a light beam (which can be conducted by optical fibers), and an optical or video vision system (such as a camera) positioned at the end present in the organ to be inspected.

[0083] The endoscope is intended to analyze human or animal biological tissues, and to be inserted into cavities or hollow organs such as a stomach, a bladder, or even a colon.

[0084] The LED source can be introduced into the working channel of the endoscope via an optical fiber. Thus, the LED source can be added to a large number of endoscopes on the market.

[0085] Preferably, the LED source used in the endoscopic imaging system generates n spectral bands having different wavelengths and preferably located between 380 and 1000 nanometers.

[0086] Each LED source preferably generates six spectral bands, with at least four spectral bands covering the visible spectrum and at least one spectral band covering the near infrared spectrum. Thus, it is for example possible to assemble two boxes to the endoscope so as to have twelve spectral bands covering the visible spectrum and the near infrared.

[0087] The housing is positioned next to the endoscopic column of the endoscope, or alternatively on the endoscopic column, and makes it possible to program and emit light with one or more spectral bands covering defined wavelength ranges. Furthermore, the housing offers the possibility of generating one or more spectral bands with a band change frequency that can be high.

[0088] During or at the end of the implementation of the data acquisition method according to the present description, the display screen can display the images acquired by the endoscope, the reflectance cubes obtained, and all the information obtained from the method described above. The display screen can also display results of an artificial intelligence processing of the information from the reflectance cubes obtained.

[0089] Such a system then makes it possible to alert the user of the endoscope to the possible presence of one or more anomalies, particularly early ones, in the biological tissues observed.

Claims

Claims

1. Method for analyzing a multispectral video previously acquired by endoscopy from biological tissues, said multispectral video having been previously acquired using a light generating device associated with an endoscope with camera and comprising: - a light device capable of generating n successive spectral bands of different wavelengths so as to cover the visible spectrum and the near infrared and generate a multispectral video in which each image corresponds to one of said spectral bands; said method comprising: - cutting said multispectral video thus obtained, so as to extract images of spectral bands of different wavelengths; - registering said images of spectral bands to obtain at least one registered multispectral image comprising an image for each spectral band of wavelengths;- the spectral reconstruction of a reflectance cube, from said recalibrated multispectral image, in which each pixel corresponds to a vector reflecting the reflectance of said area of ​​biological tissue.;

2. A method according to claim 1, wherein the wavelength bands are between 380 and 1000 nm, preferably between 400 and 850 nm.

3. A method according to any preceding claim, wherein n is six or a multiple of six.

4. The method of claim 3, wherein at least four of said spectral bands have wavelengths spanning the visible spectrum and at least one of said spectral bands has wavelengths spanning the near infrared.

5. The method of claim 4, wherein at least four of said spectral bands have wavelengths spanning the visible spectrum and at least two of said spectral bands have wavelengths spanning the near infrared.

6. A method according to any preceding claim, wherein the camera captures 20 to 30 images per second during the acquisition of the multispectral video, and wherein only one image out of two is alternately selected for the production of a multispectral image during the cutting of said obtained multispectral video.

7. A method according to any one of claims 1 to 6, wherein spectrally reconstructing a reflectance cube from said multispectral image comprises processing said multispectral image by a deep neural network.

8. Method according to any one of claims 1 to 6, wherein the spectral reconstruction of a reflectance cube from said registered multispectral images comprises: - a second registration of said registered multispectral images so as to obtain a mosaic of multispectral images; - a processing of said mosaic of multispectral images by a deep neural network or by a light-tissue interaction model.

9. A method according to any one of claims 7 and 8, wherein the deep neural network is pre-trained from reflectance spectra data characteristic of healthy and pathological tissues.

10. A method according to any one of claims 1 to 9, wherein it further comprises analyzing the reflectance cube by artificial intelligence.

11. Endoscopic imaging system characterized in that it comprises: - an endoscope configured to capture a video stream, - at least one box generating light from an LED source, and - a display screen configured to display images acquired by the endoscope and / or all or part of the information contained in reflectance cubes obtained with a data acquisition method as defined according to any one of claims 1 to 10.

Images