Endoscopic lesion absolute size measurement system and method

Abstract

The present application relates to the field of endoscopes, and discloses a system and method for measuring the absolute size of lesions under endoscopes, comprising: a light box, an illumination path and a structured light path; a light source is arranged in the light box; the illumination path comprises a shielding lens, a first coupler, a first optical fiber and an illumination lens; the shielding lens is configured to cut in and out between the first coupler and the light source; one end of the first optical fiber is connected to the first coupler, and the other end is connected to the illumination lens; the structured light path comprises a structured light generator, a second coupler, a second optical fiber and a structured light projector; the structured light generator is configured to cut in and out between the second coupler and the light source; one end of the second optical fiber is connected to the second coupler, and the other end is connected to the structured light projector.

Description

Technical Field

[0001] The present application relates to the field of endoscopy, and in particular to a system and method for measuring the absolute size of lesions under endoscopy. Background Art

[0002] The front end of the traditional endoscope in the prior art mainly includes an air and water supply nozzle, an objective lens, an illumination lens, and an instrument working channel. The illumination lens is used for illumination, and the objective lens is used for image acquisition. Generally, a gastroscope has two illumination points, and a colonoscope has three illumination points. The light source is generated by a light box, passes through the light box to a fiber bundle coupler, and then is transmitted by a light guide to the illumination lens at the front end of the endoscope for illumination.

[0003] In the application of endoscopic detection of polyps, accurate measurement of polyp size is very important for the screening and monitoring of colorectal cancer (CRC). Adenoma size is an important long-term predictive factor for CRC. Subjective polyp size measurement is of great significance for CRC risk stratification and follow-up interval. Adenomas greater than or equal to 1 cm have a shorter follow-up interval. In addition, the size of the tumor under colonoscopy is closely related to the judgment of the nature of the tumor and the choice of treatment plan. For polyps larger than 3 cm, endoscopic resection is risky and requires intestinal segment resection; however, for colon polyps smaller than 3 cm, they can be completely removed by endoscopic submucosal resection (EMR) or endoscopic submucosal dissection (ESD), and the complication rate is low. Even for polyps smaller than 3 cm, the endoscopic treatment methods for polyps of different sizes are not the same. Therefore, many guidelines use polyp size as an important indication indicator when formulating endoscopic treatment of colon polyps. However, there is no unified standard for endoscopic polyp size estimation. In clinical practice, the size of polyps is often determined by physicians based on their own experience, which is sometimes far from the actual size of polyps. Eichenseer et al. compared the size of 10-25 mm polyps estimated by 15 endoscopic experts with the size directly measured by resected tissue specimens, and found that the average variation rate was 73.6% (13%-127%) and the error rate was 62.6% (0-91%). Overestimation was more common, which led to inappropriate colorectal cancer screening recommendations for some patients.

[0004] In another study published in the journal Gastroenterology in 2016, completely resected polyps with both direct endoscopic assessment and post-resection pathological measurements were included. The authors compared the polyp measurement methods of endoscopists and pathologists and analyzed endoscopic or polyp-related factors that caused incorrect measurements. In this study, polyps were immediately immersed in formalin and sent to the pathology department for further processing after resection. The pathological size measured by a ruler with a minimum scale of mm was the gold standard. Polyps were divided into two groups, larger than 1 cm and smaller than 1 cm. Polyps were also divided into overestimated (endoscopic estimate was too high) and underestimated (endoscopic estimate was too low) groups based on patient sex, polyp morphology, histology, and polyp location. During the study period, 50 endoscopists and 12 endoscopic assistants performed a total of 9146 colonoscopies and removed a total of 6067 polyps, of which 1528 lesions from 1422 patients met the inclusion criteria. These endoscopists had an average of 15 years of endoscopic experience. The researchers found that endoscopic polyp estimates were mostly concentrated in 1 cm and 2 cm, and most of the estimates were higher than the pathological measurements. The above findings suggest that endoscopists tend to round off the polyp size to a larger measurement, which is more common in large polyps. The larger estimate will shorten the follow-up interval. About half (46%) of the polyps with an endoscopic estimate of more than 1 cm in diameter are actually less than 1 cm in size. The follow-up time of these patients may be shortened from the more appropriate 5 to 10 years to 3 years. In contrast, a small number of patients with lower estimates received inappropriately extended follow-up time. The researchers pointed out that if the estimated size of the polyp is used as the only criterion for assessing the progression of adenoma, about 10% of patients will receive unreasonable follow-up.

[0005] At present, some methods for measuring the size of polyps under endoscopy mainly include direct visual observation by endoscopists, or estimation using some instruments as references. For example, the opening size of the biopsy forceps is used as a reference, or the scales on the connecting lines of some instruments are used for measurement and estimation. However, this may lead to low measurement accuracy due to the different distances between the tool and the lesion and the different sizes and shapes of the lesions at different distances and angles.

[0006] There is a great need for an endoscopic system that can accurately measure the size of polyps. Summary of the invention

[0007] The purpose of the present application is to provide an endoscopic lesion absolute size measurement system that can automatically and accurately measure the size of polyps in an endoscope.

[0008] The embodiments of the present application provide an endoscopic lesion absolute size measurement system, comprising: a light box, an illumination pathway, and a structured light pathway;

[0009] A light source is arranged in the light box;

[0010] The lighting path includes a shielding lens, a first coupler, a first optical fiber and an illumination lens; the first coupler is configured to achieve coupling between the light box and the first optical fiber; the shielding lens is configured to cut in and out between the first coupler and the light source; one end of the first optical fiber is connected to the first coupler, and the other end is connected to the illumination lens;

[0011] The structured light path includes a structured light generator, a second coupler, a second optical fiber and a structured light projector; the second coupler is configured to achieve coupling between the light box and the second optical fiber; the structured light generator is configured to cut in and out between the second coupler and the light source; one end of the second optical fiber is connected to the second coupler, and the other end is connected to the structured light projector;

[0012] The illumination lens and the structured light projector are arranged at the end of the endoscope.

[0013] In a preferred example, the structured light generator and the shielding lens are cut in and out simultaneously.

[0014] In a preferred embodiment, when switching to the structured light mode, the shielding lens is inserted between the first coupler and the light source to block the lighting path; the structured light generator is inserted between the second coupler and the light source to generate structured light;

[0015] When switching to the white light mode, the shielding lens and the structured light generator are cut out at the same time, and the first optical fiber and the second optical fiber both transmit illumination light.

[0016] In a preferred example, the structured light generator generates visible light with a wavelength between 400-700 nm.

[0017] In a preferred example, the passband width of the light generated by the structured light generator is less than 5% of the central wavelength value.

[0018] In a preferred example, the end of the endoscope is also provided with a nozzle, an objective lens and an instrument outlet.

[0019] In a preferred example, the light source is a xenon lamp, a laser or an LED.

[0020] The present application also discloses a method for measuring the absolute size of a lesion under endoscopy, comprising the following steps:

[0021] As described above, the endoscopic lesion absolute size measurement system is used to capture images in white light mode, and the captured images are input into a trained object detection model to detect the presence of polyps in real time;

[0022] If a polyp is detected, a deep neural network based on an encoder-decoder structure is used to automatically segment the polyp area in real time;

[0023] Switching the endoscopic lesion absolute size measurement system to a structured light mode to perform structured light imaging;

[0024] The collected structured light images were analyzed using an image processing system, and the polyp size was calculated based on the polyp segmentation results.

[0025] In a preferred example, the target detection model includes a first encoder and a detector;

[0026] The first encoder is configured to perform feature extraction on an image obtained in a white light mode to obtain a feature map;

[0027] The detector is configured to regress the input feature map to obtain the coordinates of the location of the polyp.

[0028] In a preferred embodiment, the deep neural network based on the encoder-decoder structure includes:

[0029] A second encoder for inputting an image, comprising a convolutional layer, an activation layer, and a pooling layer;

[0030] A decoder for outputting results, including convolutional layers, activation layers, concatenation layers, and upsampling layers;

[0031] The second encoder and the decoder fuse feature maps whose size differences are less than a predetermined threshold through cross-layer feature fusion.

[0032] The endoscopic lesion absolute size measurement system and method of the present application have the following technical effects:

[0033] The endoscopic lesion absolute size measurement system and method of the present application can accurately measure the size of polyps, so that users can reasonably stratify the patient's CRC risk and reasonably arrange the follow-up interval time, and provide appropriate colorectal tumor screening recommendations.

[0034] The specification of the present application records a large number of technical features, which are distributed in various technical solutions. If all possible combinations of technical features of the present application (i.e., technical solutions) are to be listed, the specification will be too long. In order to avoid this problem, the various technical features disclosed in the above-mentioned invention content of the present application, the various technical features disclosed in the various embodiments and examples below, and the various technical features disclosed in the accompanying drawings can be freely combined with each other to form various new technical solutions (these technical solutions should all be regarded as having been recorded in this specification), unless the combination of such technical features is technically infeasible. For example, in one example, feature A+B+C is disclosed, and in another example, feature A+B+D+E is disclosed, and features C and D are equivalent technical means that play the same role. Technically, only one can be used, and it is impossible to use them at the same time. Feature E can be combined with feature C technically. Then, the solution of A+B+C+D should not be regarded as having been recorded because it is technically infeasible, and the solution of A+B+C+E should be regarded as having been recorded. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 is a schematic structural diagram of an endoscopic lesion absolute size measurement system according to the present application;

[0036] Figure 2 is a schematic flow chart of a method for measuring absolute size of lesions under endoscopy according to the present application;

[0037] Figure 3 is a structural schematic diagram of a target detection model according to the method for measuring absolute size of lesions under endoscopy of the present application;

[0038] Figure 4 It is a schematic diagram of the structure of a U-NET network based on an encoder-decoder structure according to the method for measuring the absolute size of lesions under endoscopy of the present application;

[0039] Figure 5 is a schematic diagram of a lesion size calculation model according to the endoscopic lesion absolute size measurement method of the present application;

[0040] Description of reference numerals:

[0041] 1. Light source,

[0042] 2. Structured light path,

[0043] 3. Lighting pathways,

[0044] 4. Block the lens,

[0045] 5. The first coupler,

[0046] 6. First optical fiber,

[0047] 7. Lighting lens,

[0048] 8. Structured light generator,

[0049] 9. Second coupler,

[0050] 10. Second optical fiber,

[0051] 11. Structured light projector,

[0052] 12. Nozzle,

[0053] 13. Objective lens,

[0054] 14. Export of equipment,

[0055] 15. Light box. DETAILED DESCRIPTION

[0056] In the following description, many technical details are provided to help readers better understand the present application. However, those skilled in the art can understand that the technical solution claimed in the present application can be implemented even without these technical details and various changes and modifications based on the following embodiments.

[0057] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below in conjunction with the accompanying drawings.

[0058] The first embodiment of the present application relates to an endoscopic lesion absolute size measurement system, such as Figure 1 As shown, the endoscopic lesion absolute size measurement system includes: a light box, an illumination path and a structured light path;

[0059] A light source is arranged in the light box;

[0060] The lighting path includes a shielding lens, a first coupler, a first optical fiber and an illumination lens; the first coupler is configured to achieve coupling between the light box and the first optical fiber; the shielding lens is configured to cut in and out between the first coupler and the light source; one end of the first optical fiber is connected to the first coupler, and the other end is connected to the illumination lens;

[0061] The structured light path includes a structured light generator, a second coupler, a second optical fiber and a structured light projector; the second coupler is configured to achieve coupling between the light box and the second optical fiber; the structured light generator is configured to cut in and out between the second coupler and the light source; one end of the second optical fiber is connected to the second coupler, and the other end is connected to the structured light projector;

[0062] The illumination lens and structured light projector are arranged at the end of the endoscope.

[0063] Optionally, in one embodiment, the structured light passes through a coupler and a structured light optical fiber, and is then projected onto the tissue surface by a structured light projector. After being projected onto the surface of the object to be measured, the structured light is modulated by the height of the object to be measured, and the modulated structured light is collected by a camera system and transmitted to a computer for analysis and calculation to obtain three-dimensional data of the object to be measured.

[0064] The structured light generator and the occluding lens are cut in and out at the same time.

[0065] When switched to the structured light mode, the shielding lens is inserted between the first coupler and the light source to block the lighting path; the structured light generator is inserted between the second coupler and the light source to generate structured light;

[0066] When switched to the white light mode, the shielding lens and the structured light generator are cut out at the same time, and both the first optical fiber and the second optical fiber transmit illumination light.

[0067] Optionally, in one embodiment, the structured light generator can be switched by a button or system control. If the structured light generator is switched on, the rear end of the illumination light guide block the lens and switch on at the same time, preventing the illumination light source from passing through, and only the structured light projector projects the structured light. The collected structured light image is sent to the host for processing and analysis of the three-dimensional data contained therein; if the structured light generator is switched off, the rear end of the illumination light guide block the lens and switch on at the same time, allowing the illumination light source to pass through and perform white light illumination, and the structured light path is used as a general illumination path for white light illumination.

[0068] The structured light generator generates visible light with a wavelength between 400-700nm or a passband width of the generated light that is less than 5% of the central wavelength value. The actual selection will be determined through experiments to minimize reflections to obtain the clearest structured light image.

[0069] Optionally, in one embodiment, the end of the endoscope is also configured with a nozzle, an objective lens and an instrument outlet.

[0070] Optionally, in one embodiment, the light source is a xenon lamp, a laser or an LED.

[0071] The second embodiment of the present application relates to a method for measuring the absolute size of a lesion under endoscopy, comprising the following steps: Figure 2 As shown:

[0072] In step 202, the endoscopic lesion absolute size measurement system described in the first embodiment is used to perform photography in white light mode, and the photographed images are input into a trained target detection model to detect in real time whether polyps are present.

[0073] If a polyp is detected, the process proceeds to step 204 to perform real-time automatic segmentation of the polyp region using a deep neural network based on an encoder-decoder structure.

[0074] Then, the process proceeds to step 206 , in which the endoscopic lesion absolute size measurement system is switched to a structured light mode to perform structured light imaging.

[0075] Then, the process proceeds to step 208 , where the collected structured light image is analyzed using an image processing system, and the polyp size is calculated in combination with the polyp segmentation result.

[0076] In a preferred example, the target detection model includes a first encoder and a detector; the first encoder is configured to perform feature extraction on an image obtained in a white light mode to obtain a feature map; the detector is configured to regress the input feature map to obtain the coordinates of the location of the polyp.

[0077] In a preferred embodiment, the deep neural network based on the encoder-decoder structure includes:

[0078] A second encoder for inputting an image, comprising a convolutional layer, an activation layer, and a pooling layer;

[0079] A decoder for outputting results, including convolutional layers, activation layers, concatenation layers, and upsampling layers;

[0080] The second encoder and decoder fuse feature maps whose size differences are less than a predetermined threshold through cross-layer feature fusion.

[0081] Optionally, in one embodiment, Figure 3 The target detection model shown in the figure takes the image x under white light as input, and inputs an encoder for feature extraction. The feature map obtained after feature extraction is then regressed through a detector to obtain the coordinate y of the location of the polyp. The encoder and detector are composed of a deep neural network, including convolution layer, downsampling layer, upsampling layer, pooling layer, batch normalization layer, activation layer, etc. Through the above target detection model, when the system detects the presence of polyps, the system will prompt the doctor with a red box on the monitor to indicate the rectangular area where the polyps are located, and sound an alarm. The system then performs fine segmentation of the polyp area.

[0082] The fine segmentation of the polyp area is based on a deep neural network with an encoder-decoder structure. The left half is the encoder, which includes convolutional layers, activation layers, and pooling layers. The right half is the decoder, which includes convolutional layers, activation layers, concatenation layers, and upsampling layers. The encoder and decoder directly fuse feature maps of similar sizes through cross-layer feature fusion, which can better extract contextual information and global information. Figure 4 Shown is a U-NET network based on an encoder-decoder structure.

[0083] Once the lesion is segmented, the system automatically or manually switches to the structured light imaging mode, and the rear end of the lighting pathway is closed by blocking the lens to prevent the passage of the lighting source. The structured light is projected onto the tissue surface through the above structured light pathway, and the deformation of the structured light image will be collected by the endoscope camera and transmitted to the image processing center for processing. After the collected structured light image is transmitted to the image processing center, the image processing center combines the segmentation result with the deformation of the structured light image, calculates the three-dimensional contour of the lesion through the phase shift method, and gives the lesion size.

[0084] Optionally, in one embodiment, Figure 5 As shown in the figure, a model for calculating the size of lesions is shown. P is the optical center of the projector, and C is the optical center of the camera. O is the intersection of the optical axis of the camera and the optical axis of the projector. Let the horizontal plane passing through point O be the reference X-axis in the calculation. L1 and L2 are the distances from the optical center of the camera and the optical center of the projector to the X-axis respectively. d is the distance from the optical center of the camera to the optical center of the projector along the X-axis. The ABO plane is a hypothetical virtual plane, and the hypothetical virtual plane is parallel to the connecting line PC of the optical center of the projector and the optical center of the camera. For a point Q on the surface of the object, the height Z of point Q relative to the reference plane can be solved by the trigonometric relationship:

[0085]

[0086] Since the projection from plane to plane is a linear mapping, the projector pattern mapping on the imaginary plane ABO also has a fixed period. Into a linear relationship.

[0087] Then calculate the absolute phase of point Q When the sinusoidal fringe image is projected onto a three-dimensional diffuse reflective surface, the image of a point Q(x, y) observed from the camera can be expressed as:

[0088]

[0089] Where A(x,y) is the background light intensity, B(x,y) / A(x,y) represents the contrast of the grating fringes, is the phase value. For the N-step phase shift method, the phase difference between each step of the sinusoidal grating phase shift and the projected phase of the previous step of the sinusoidal grating phase shift is 2π / N, that is, the following equation group:

[0090]

[0091] Where i=1,2,…,N. The system of equations has N equations and three unknowns. It can be solved when N>=3. The solution is:

[0092]

[0093]

[0094] Through phase unwrapping, the absolute phase value of point Q (x, y) can be obtained Will Substituting the above formula, we can obtain the height Z of point Q from the imaginary plane ABO.

[0095] By calibrating the camera parameters, the absolute coordinates (X, Y, Z) of point Q in the real world can be calculated based on Z. By calculating the absolute coordinates (X, Y, Z) of each point on the collected structured light image and combining it with the above polyp segmentation results, the three-dimensional size of the polyp can be obtained.

[0096] It should be noted that in the application documents of this patent, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "including one" do not exclude the existence of other identical elements in the process, method, article or device including the elements. In the application documents of this patent, if it is mentioned that an action is performed according to an element, it means that the action is performed at least according to the element, which includes two situations: performing the action only according to the element, and performing the action according to the element and other elements. Expressions such as multiple, multiple, and multiple include 2, 2 times, 2 kinds, and more than 2, more than 2 times, and more than 2 kinds.

[0097] This specification includes combinations of the various embodiments described herein. Individual references to "one embodiment" or a particular embodiment, etc., do not necessarily refer to the same embodiment; however, these embodiments are not mutually exclusive unless indicated as mutually exclusive or clear to those skilled in the art. It should be noted that the word "or" is used in this specification in a non-exclusive sense unless the context clearly indicates or requires otherwise.

[0098] All documents mentioned in this application are considered to be included in the disclosure of this application as a whole, so that they can be used as the basis for modification when necessary. In addition, it should be understood that after reading the above disclosure of this application, those skilled in the art can make various changes or modifications to this application, and these equivalent forms also fall within the scope of protection claimed in this application.

Claims

1. An endoscopic lesion absolute size measurement system, characterized in that: include: light boxes, lighting pathways, and structured light pathways; A light source is arranged in the light box; The lighting path includes a shielding lens, a first coupler, a first optical fiber and an illumination lens; the first coupler is configured to achieve coupling between the light box and the first optical fiber; the shielding lens is configured to cut in and out between the first coupler and the light source; One end of the first optical fiber is connected to the first coupler, and the other end is connected to the illumination lens; The structured light path includes a structured light generator, a second coupler, a second optical fiber and a structured light projector; the second coupler is configured to achieve coupling between the light box and the second optical fiber; the structured light generator is configured to cut in and out between the second coupler and the light source; One end of the second optical fiber is connected to the second coupler, and the other end is connected to the structured light projector; When switching to the white light mode, the shielding lens and the structured light generator are cut out at the same time, and the first optical fiber and the second optical fiber both transmit illumination light; Shooting in the white light mode, inputting the shot images into a trained target detection model to detect in real time whether polyps are present; if polyps are detected, using a deep neural network based on an encoder-decoder structure to automatically segment the polyp area in real time; When switching to the structured light mode, the shielding lens cuts into between the first coupler and the light source to shield the lighting path; The structured light generator is inserted between the second coupler and the light source to generate structured light; the structured light can be modulated by the height of the object to be measured after being projected onto the surface of the object to be measured by the structured light projector, and the modulated structured light can be collected by the camera system to calculate the three-dimensional data of the object to be measured; The illumination lens and the structured light projector are arranged at the end of the endoscope.

2. The endoscopic lesion absolute size measurement system according to claim 1, characterized in that: The structured light generator and the shielding lens are cut in and out simultaneously.

3. The endoscopic lesion absolute size measurement system according to claim 2, characterized in that: in, The three-dimensional data of the object to be measured is calculated in the following way: P is the optical center of the projector, C is the optical center of the camera; O is the intersection of the optical axis of the camera and the optical axis of the projector; the horizontal plane passing through point O is assumed to be the reference X-axis in the calculation; L1 and L2 are the distances from the optical center of the camera and the optical center of the projector to the X-axis respectively; d is the distance from the optical center of the camera to the optical center of the projector along the X-axis; the ABO plane is a hypothetical virtual plane, which is parallel to the connecting line PC of the optical center of the projector and the optical center of the camera; for a point Q on the surface of the object, the height Z of point Q relative to the reference plane can be solved by trigonometric relationship: Since the projection from plane to plane is a linear mapping, the projector pattern mapping on the imaginary plane ABO also has a fixed period. Into a linear relationship; Then calculate the absolute phase of point Q When the sinusoidal fringe image is projected onto a three-dimensional diffuse reflective surface, the image of a point Q(x, y) observed from the camera can be expressed as: Where A(x,y) is the background light intensity, B(x,y) / A(x,y) represents the contrast of the grating fringes, is the phase value; for the N-step phase shift method, the phase difference between each step of the sinusoidal grating phase shift and the projected phase of the previous step of the sinusoidal grating phase shift is 2π / N, that is, the following equation group: Where i=1,2,…,N; the system of equations has N equations and three unknowns, and can be solved when N>=3; the solution is: Through phase unwrapping, the absolute phase value of point Q (x, y) can be obtained Will Substituting into the above formula, we can get the height Z of point Q from the imaginary plane ABO; By calibrating the camera parameters, the absolute coordinates (X, Y, Z) of point Q in the real world are calculated based on Z.

4. The endoscopic lesion absolute size measurement system according to claim 1, characterized in that: The structured light generator generates visible light with a wavelength between 400-700nm.

5. The endoscopic lesion absolute size measurement system according to claim 1, characterized in that: The passband width of the light generated by the structured light generator is less than 5% of the central wavelength value.

6. The endoscopic lesion absolute size measurement system according to claim 1, characterized in that: The end of the endoscope is also configured with a nozzle, an objective lens and an instrument outlet.

7. The endoscopic lesion absolute size measurement system according to claim 1, characterized in that: The light source is a xenon lamp, a laser or an LED.

Images