Shape measurement system for endoscope and shape measurement method for endoscope

Abstract

The display processing section (46) displays an endoscope image of a biological tissue in a body cavity photographed by an endoscope on a display device. The operation accepting section (30) accepts a user operation for setting a region of interest in the endoscope image. The region of interest setting section (34) sets the region of interest in the endoscope image in accordance with the user operation. The three-dimensional information acquiring section (38) acquires three-dimensional shape information of the biological tissue photographed by the endoscope. The virtual surface deriving section (48) derives three-dimensional shape information of a virtual surface in the region of interest in accordance with the three-dimensional shape information of a region different from the region of interest. The size information determining section (50) determines information about a size of the virtual surface in accordance with the three-dimensional shape information of the virtual surface.

Description

Technical Field

[0001] This invention relates to a shape measurement system and a shape measurement method for endoscopes. Background Technology

[0002] Endoscopic submucosal dissection (ESD) is a treatment method that uses specialized instruments to remove lesions simultaneously, replacing traditional surgical treatments and gaining attention as a less invasive approach. Tumors targeted for ESD are those without the risk of lymph node metastasis. Metastasis rates are related to tumor size; therefore, accurate measurement of tumor size is crucial.

[0003] Patent Document 1 discloses a three-dimensional shape measuring device that projects a laser-based measurement pattern onto an observation area, and calculates the three-dimensional shape of the observation area based on the image of the measurement pattern projected onto the observation area. Patent Document 2 discloses a technique for determining a reference surface by determining the three-dimensional coordinates of multiple points on the surface of an object close to anomalies such as dents, cracks, or pitting.

[0004] Existing technical documents

[0005] Patent documents

[0006] Patent Document 1: Japanese Patent Application Publication No. 2017-23562

[0007] Patent Document 2: Japanese Patent Application Publication No. 2017-162452 Summary of the Invention

[0008] The problem that the invention aims to solve

[0009] Not only is accurate measurement of the size of lesions such as tumors important as a criterion for evaluating the implementation of endoscopy-based surgical procedures (ESDs), but it is also crucial in endoscopic diagnosis. However, the inner lining of the digestive tract is curved, and previously, no method had been established to accurately measure the size of lesions present on such an inner lining. This invention was made in view of this situation, and its object is to provide a technique for accurately determining information related to the size of lesions contained in endoscopic images.

[0010] means for solving problems

[0011] To address the aforementioned problems, one aspect of the endoscopic shape measurement system of the present invention includes a processor. The processor displays endoscopic images of biological tissue within a body cavity captured by the endoscope on a display device, accepts user input to set a region of interest in the endoscopic image, sets the region of interest in the endoscopic image based on the user input, acquires three-dimensional shape information of the biological tissue captured by the endoscope, derives three-dimensional shape information of an imaginary surface within the region of interest based on the three-dimensional shape information of a region different from the region of interest, and determines information related to the size of the imaginary surface based on the derived three-dimensional shape information of the imaginary surface. The endoscopic shape measurement system may have multiple processors, which cooperate to perform the above processing.

[0012] Another aspect of the present invention uses an endoscopic shape measurement method to display endoscopic images of biological tissue within a body cavity captured by an endoscope on a display device. It accepts user input to set a region of interest in the endoscopic image. Based on the user input, the region of interest is set in the endoscopic image, and three-dimensional shape information of the biological tissue captured by the endoscope is obtained. Based on the three-dimensional shape information of a region different from the region of interest, the three-dimensional shape information of an imaginary surface within the region of interest is derived. Based on the derived three-dimensional shape information of the imaginary surface, information related to the size of the imaginary surface is determined.

[0013] Furthermore, any combination of the above-mentioned constituent elements, or any way in which the present invention is embodied in a method, apparatus, system, recording medium, computer program, etc., is also valid as a way of the present invention. Attached Figure Description

[0014] Figure 1 (a) is a diagram showing an example of an image captured by an endoscope, and (b) is a diagram schematically showing the situation during the capture of an endoscopic image.

[0015] Figure 2 This is a diagram showing the structure of a shape measurement system for endoscopes.

[0016] Figure 3 This is a diagram showing the functional blocks in an endoscopic shape measurement system.

[0017] Figure 4 This is a diagram showing an example of an endoscopic image.

[0018] Figure 5 This is a flowchart illustrating the measurement process for the shape of a lesion.

[0019] Figure 6 It is a diagram representing the defined area of ​​interest.

[0020] Figure 7 It is a diagram representing the set reference area.

[0021] Figure 8 It is a diagram that shows the state of the reference region being configured in the xy plane.

[0022] Figure 9 (a) and (b) are figures representing examples of the three-dimensional shape of the reference region.

[0023] Figure 10 This is a diagram used to illustrate the method for deriving the three-dimensional shape of an imaginary surface.

[0024] Figure 11 (a) and (b) are diagrams representing the three-dimensional shape of the derived imaginary surface.

[0025] Figure 12 Figures (a) to (c) are used to illustrate the method for calculating the size of an imaginary surface.

[0026] Figure 13 This is an example image showing a confirmation screen for the area of ​​interest.

[0027] Figure 14 This is a diagram illustrating an example of a reference region containing discontinuities.

[0028] Figure 15 It is a diagram representing the defined exclusion area. Detailed Implementation

[0029] Figure 1 (a) represents an example of an endoscopic image of biological tissue within a body cavity. In this endoscopic image, lesion 2 is contained in the central region, surrounded by a biological surface 3 that serves as the inner wall of the digestive tract. Figure 1 (b) schematically illustrates the situation during endoscopic imaging. In this example, lesion 2 is a tumor that protrudes from the curved biological surface 3, but there is also a tumor in lesion 2 that is recessed from the biological surface 3.

[0030] Because the biological surface 3 has a curved shape, the length L between point A and point B in lesion 2 is not the length of the line segment connecting point A and point B, but rather the length along the curved biological surface 3. The endoscopic image does not include the biological surface 3 obscured by lesion 2; therefore, in this embodiment, the three-dimensional shape of the biological surface 3 obscured by lesion 2 (hereinafter referred to as the "imaginary surface") is estimated, and the size of lesion 2 is measured.

[0031] Figure 2The structure of the endoscopic shape measurement system 1 according to the embodiment is shown. The endoscopic shape measurement system 1 is installed in medical facilities such as hospitals where endoscopic examinations are performed. In the endoscopic shape measurement system 1, the endoscopic observation device 5, the information processing device 10a, the information processing device 10b, and the image analysis device 8 are connected in a communicable manner via a network 4 such as a LAN (local area network).

[0032] An endoscopic observation device 5 is installed in the examination room and connected to an endoscope 7 inserted into the patient's digestive tract. The endoscope 7 has a light guide for illuminating the digestive tract by transmitting illumination light supplied from the endoscopic observation device 5. At its front end, it has an illumination window for projecting the illumination light transmitted by the light guide onto the biological tissue, and an imaging unit that takes images of the biological tissue at predetermined intervals and outputs the image signals to the endoscopic observation device 5. The endoscopic observation device 5 supplies illumination light to the endoscope 7 corresponding to the observation mode. The imaging unit includes a solid-state imaging element (e.g., a CCD image sensor or a CMOS image sensor) that converts incident light into electrical signals.

[0033] The endoscopic observation device 5 performs image processing on the image signal obtained by photoelectric conversion from the solid-state imaging element of the endoscope 7 to generate an endoscopic image, which is then displayed in real time on the display device 6. In addition to conventional image processing functions such as A / D conversion and noise removal, the endoscopic observation device 5 can also perform special image processing for purposes such as highlighting the image. By equipping the endoscopic observation device 5 with special image processing functions, it can generate both unprocessed and specially processed endoscopic images based on image signals captured using the same illumination light.

[0034] In this embodiment, the endoscopic image can be a WLI (White Light Imaging) observation image generated from a camera signal captured using ordinary light (white light), a TXI (Texture and Color Enhancement Imaging) observation image generated by performing special image processing on a camera signal captured using ordinary light, an RDI observation image generated from a camera signal captured using special light, an NBI (Narrow Band Imaging) observation image, or an AFI (Autofluorescence Imaging) observation image. Furthermore, the endoscopic image can be a pseudo-colorized image of the subject's contours, or it can be an image generated by processing the camera signal from the endoscope 7.

[0035] The doctor observes the endoscopic images displayed on the display device 6 according to the examination procedures. When the doctor operates the release switch of the endoscope 7, the endoscopic observation device 5 captures (saves) the endoscopic image at the moment the release switch is operated, and sends the captured endoscopic image to the image storage server 9. The endoscopic images stored in the image storage server 9 are used by the doctor to generate an examination report.

[0036] The information processing device 10a is installed in the examination room to allow users such as doctors or nurses to review information related to the size of lesions in endoscopic images during endoscopic examinations in real time. The information processing device 10a can cooperate with the image analysis device 8 to provide users with information related to the size of lesions.

[0037] The information processing device 10b is located in a room outside the examination room and is used by the doctor when generating examination reports. For example, to confirm whether a lesion captured during a detailed examination is the size that will become an object for ESD in a next examination, the doctor can use the lesion shape measurement function of the information processing device 10b. Once the doctor confirms that the captured lesion is the size that will become an object for ESD, they can decide to use ESD to remove the lesion during the next endoscopic examination.

[0038] The image analysis device 8 has the function of detecting lesions and outputting the region where the lesion exists (lesion area) when an endoscopic image is input. The image analysis device 8 can utilize a learned model, which is generated by machine learning using training data of endoscopic images and information related to the lesion area contained in the endoscopic image. When an endoscopic image is input, the learned model outputs the location of the lesion area. During endoscopic examination, the image analysis device 8 can receive endoscopic images from the endoscopic observation device 5 and provide the image analysis results to the endoscopic observation device 5 or the information processing device 10a.

[0039] Figure 3 This represents the functional blocks in the endoscopic shape measurement system 1. The endoscopic shape measurement system 1 includes a processing unit 20 for performing lesion shape measurement processing, a learned model holding unit 60, and a user interface 70. The processing unit 20 includes an operation receiving unit 30, a region setting unit 32, a three-dimensional information acquisition unit 38, an image generation unit 40, an image analysis unit 42, an auxiliary information generation unit 44, a display processing unit 46, an imaginary surface derivation unit 48, and a size information determination unit 50. The region setting unit 32 includes a region of interest setting unit 34 and a reference region setting unit 36. The user interface 70 is, for example, a mouse, stylus, keyboard, or other tool for user input.

[0040] Figure 3The structure shown can be implemented in hardware using any processor, memory, auxiliary storage device, or other LSI, and in software using a program loaded into memory, etc., but functional blocks implemented through their cooperation are depicted herein. Therefore, those skilled in the art will understand that these functional blocks can be implemented in various forms, either in hardware only, software only, or a combination thereof.

[0041] Figure 3 The functional blocks of the endoscopic shape measurement system 1 shown are implemented by at least one of the following devices: endoscopic observation device 5, information processing device 10a, information processing device 10b, and image analysis device 8. The functional blocks of the endoscopic shape measurement system 1 can be implemented individually by the endoscopic observation device 5, information processing device 10a, information processing device 10b, and image analysis device 8, or they can be implemented by a combination of two or more devices. Therefore, the functional blocks shown as processing devices 20 can be implemented by one or more processors included in a single device, or they can be implemented by multiple processors included in two or more devices.

[0042] During an endoscopic examination, the endoscopic observation device 5 displays images of the digestive tract captured by the endoscope 7 on the display device 6. The physician moves the endoscope 7 while observing the endoscopic images displayed on the display device 6, and when a lesion appears on the display device 6, operates the release switch of the endoscope 7. The endoscopic observation device 5 captures an endoscopic image at the moment the release switch is operated and sends the captured endoscopic image, along with information identifying the endoscopic image (image ID), to the image storage server 9. Alternatively, the endoscopic observation device 5 can also send multiple captured endoscopic images to the image storage server 9 after the examination. The image storage server 9 records the endoscopic images sent from the endoscopic observation device 5 in association with the examination ID that identifies the endoscopic examination.

[0043] The endoscopic observation device 5 of this embodiment has the function of measuring the three-dimensional shape information of biological tissue contained in the captured endoscopic images. The endoscopic observation device 5 can measure the three-dimensional shape of the captured biological tissue using known methods. The endoscopic observation device 5 measures the three-dimensional shape of the biological tissue contained in the captured endoscopic images, appends the measured three-dimensional shape information to the endoscopic images, and sends it to the image accumulation server 9. Therefore, the image accumulation server 9 records the captured three-dimensional shape information of the biological tissue in association with the endoscopic images.

[0044] For example, the endoscope 7 is equipped with a stereo camera, and the endoscopic observation device 5 can measure the three-dimensional shape of biological tissue based on the principle of triangulation using images captured by two cameras. Alternatively, as disclosed in Patent Document 1, the endoscopic observation device 5 can project a laser-based measurement pattern onto the biological tissue, and measure the three-dimensional shape of the biological tissue based on the image captured by the projected measurement pattern. Furthermore, the endoscopic observation device 5 can also use a learned model trained on images acquired by a stereo camera and the distance information of the biological tissue contained in those images as training data to measure the three-dimensional shape of the biological tissue based on endoscopic images captured by a monocular camera. Additionally, the endoscopic observation device 5 can also measure the three-dimensional shape of the biological tissue based on the inter-frame features of the endoscopic images captured by the monocular camera. In this way, the endoscopic observation device 5 uses known measurement methods to measure the three-dimensional shape of biological tissue in endoscopic images.

[0045] Furthermore, the three-dimensional shape measurement function of the captured biological tissue can also be mounted on a device other than the endoscope observation device 5. For example, the three-dimensional shape measurement function can be mounted on the information processing device 10a, the image analysis device 8, or the image accumulation server 9. The three-dimensional shape of the biological tissue can be measured by any device, and the image accumulation server 9 records the endoscopic images sent from the endoscope observation device 5 in association with the captured three-dimensional shape information of the biological tissue or the data used to calculate the three-dimensional shape information.

[0046] After the examination, the doctor operates the information processing device 10b to generate an examination report. The information processing device 10b reads the endoscopic images captured during the examination from the image storage server 9 and displays them on the display device 12b. The doctor diagnoses the lesions contained in the displayed endoscopic images. In this embodiment, the information processing device 10b, upon request from the doctor, performs a process to deduce the three-dimensional shape of the biological surface (imaginary surface) obscured by the lesion contained in the endoscopic image and determine the size of the imaginary surface, i.e., the size of the lesion's base surface. Hereinafter, the information processing device 10b implements the processing device 20 (see reference 20). Figure 3 The description of the lesion shape measurement function is explained.

[0047] Figure 4 An example of an endoscopic image displayed on display device 12b is shown. Display processing unit 46 retrieves the endoscopic image associated with the examination ID of the examination for which the report was generated from image storage server 9 and displays it on display device 12b. The endoscopic image is an image of biological tissue within a body cavity captured by endoscope 7. Figure 4 The endoscopic image shown contains a roughly circular lesion 100 and a biological surface 102 surrounding it, which serves as the inner wall of the digestive tract.

[0048] Figure 5 A flowchart illustrating the lesion shape measurement process. The doctor operates the user interface 70 to set the region of interest in the endoscopic image displayed on the display device 12b (S10).

[0049] Figure 6 An example of a defined region of interest 110 is shown below. The user operates a user interface 70, such as a mouse, to draw a boundary line between the lesion 100 and the biological surface 102 in the endoscopic image displayed on the display device 12b. When the operation receiving unit 30 receives a user operation to set a region of interest in the endoscopic image, the region of interest setting unit 34 sets the region of interest 110 in the endoscopic image according to the user operation. The region of interest 110 is the lesion area surrounded by the boundary line. Since the user, as a physician, can identify the boundary between the lesion 100 and the biological surface 102 based on medical observations such as three-dimensional shape or tonal characteristics, the region of interest setting unit 34 can accurately set the region of interest 110 representing the lesion area according to the user operation.

[0050] After the region of interest setting unit 34 sets the region of interest 110, the reference region setting unit 36 ​​sets a reference region surrounding the region of interest 110 based on the region of interest 110 (S12). The reference region 120 includes at least the entire region of interest 110 and is set to be a larger area than the region of interest 110. By setting the reference region by the reference region setting unit 36, even if the surface of the organism has a complex curved shape, as will be described later, the imaginary surface derivation unit 48 can derive the three-dimensional shape information of the imaginary surface with high accuracy.

[0051] Figure 7 This illustrates an example of a defined reference region 120. The reference region setting unit 36 ​​preferably sets the reference region 120 in such a way that it has a blank space of at least a predetermined length outside the outer edge of the region of interest 110. For example, the reference region setting unit 36 ​​may also set a rectangular region consisting of a left side, a right side, a top side, and a bottom side as the reference region 120, wherein the left and right sides are located at positions departing from the leftmost and rightmost ends of the region of interest 110 by a predetermined length l1, respectively, and the top and bottom sides are located at positions departing from the topmost and bottommost ends of the region of interest 110 by a predetermined length l2, respectively.

[0052] Alternatively, the user can operate the user interface 70 to set a reference region 120 for the area of ​​interest 110 displayed on the display device 12b. When the operation receiving unit 30 receives a user operation to set a reference region in the endoscopic image, the reference region setting unit 36 ​​sets the reference region 120 in the endoscopic image based on the user operation. The set reference region 120 is used to determine the range of three-dimensional shape information used when deriving the imaginary surface of the area of ​​interest 110, so the user preferably determines the range of the imaginary surface of the area of ​​interest 110 that can be appropriately derived as the reference region 120.

[0053] After the reference region setting unit 36 ​​sets the reference region 120, the three-dimensional information acquisition unit 38 acquires the three-dimensional shape information of the biological tissue captured by the endoscope (S14). As described above, the three-dimensional shape information of the biological tissue is recorded in the image storage server 9 in association with the endoscope image, and the three-dimensional information acquisition unit 38 acquires the three-dimensional shape information associated with the endoscope image.

[0054] The imaginary surface derivation unit 48 derives the three-dimensional shape information of the imaginary surface in the region of interest 110 based on the three-dimensional shape information of a region different from the region of interest 110 (S16). The three-dimensional shape information of the imaginary surface may include the two-dimensional coordinates of each pixel of the imaginary surface in the endoscopic image and the distance information (depth information) of each pixel. Specifically, the imaginary surface derivation unit 48 derives the three-dimensional shape information of the imaginary surface in the region of interest 110 based on the three-dimensional shape information of the reference region 120. For example... Figure 1 As shown, the imaginary surface in the region of interest 110 refers to the imaginary biological surface existing inside the lesion 100 (assumed to be the biological surface 102 when the lesion 100 does not exist).

[0055] Figure 8 This indicates the state where reference region 120 is configured in the xy plane. Figure 8 In this model, the x-coordinate represents the lateral position of the endoscopic image, and the y-coordinate represents the vertical position. Furthermore, the distance (depth) of each pixel in the 3D information from the camera can be represented by the z-coordinate. Figure 8 In the diagram, the shaded area represents the reference region 120 on the xy plane, and the imaginary surface derivation unit 48 derives the three-dimensional shape information of the imaginary surface in the region of interest 110 based on the three-dimensional shape information of the reference region 120. When deriving the three-dimensional shape information of the imaginary surface, the imaginary surface derivation unit 48 does not utilize the three-dimensional shape information of the region of interest 110.

[0056] Figure 9 An example of the three-dimensional shape of reference region 120 is shown. Figure 9 (a) is a stereoscopic view of the three-dimensional shape of reference region 120. Figure 9(b) is a diagram showing the three-dimensional shape of the reference region 120 as viewed along the y-axis.

[0057] Figure 10 This diagram illustrates a method for deriving the three-dimensional shape of an imaginary surface in the region of interest 110. The imaginary surface derivation unit 48 performs fitting processing on the three-dimensional shape of the region of interest 110 for each column in the y-axis direction, spaced at predetermined intervals along the x-axis, based on the three-dimensional shape information of the reference region 120. Specifically, as... Figure 10 As shown, for each column extending in the y-axis direction, a polynomial approximation curve is derived based on the three-dimensional point group data in the reference region 120. The imaginary surface derivation unit 48 divides the region of interest 110 into N equally spaced sections in the x-axis direction and derives a polynomial approximation curve fitted to each section (column). Then, the imaginary surface derivation unit 48 can also perform fitting processing in the x-axis direction based on the point group data of the N polynomial approximation curves to derive the three-dimensional shape information of a smooth imaginary surface. In this way, by utilizing the reference region 120 set by the reference region setting unit 36, the imaginary surface derivation unit 48 can derive the three-dimensional shape information of the imaginary surface with high accuracy, even if the surface of the organism has a complex curved shape.

[0058] Figure 11 (a) shows the target Figure 9 Example of an imaginary surface 130 derived from the three-dimensional shape of reference region 120 shown in (b), and Figure 11 (b) shows an example of the three-dimensional shape of the derived imaginary surface 130. Furthermore, the imaginary surface derivation unit 48 of this embodiment performs a fitting process for the three-dimensional shape of the derived polynomial approximation curve, but other types of fitting processes can also be performed. The imaginary surface derivation unit 48 of this embodiment smoothly connects the space vacated in the center of the three-dimensional shape of the reference region 120 to derive the imaginary surface 130 in the region of interest 110. The imaginary surface 130 thus generated corresponds to the imaginary bottom surface of the lesion 100 in the region of interest 110.

[0059] The size information determination unit 50 determines information related to the size of the imaginary surface 130 based on the three-dimensional shape information of the imaginary surface 130 (S18). For example, the size information determination unit 50 can deduce the major and minor axes of the imaginary surface 130 as follows.

[0060] Figure 12 (a) is a diagram illustrating the method for calculating the major axis of the imaginary surface 130. The size information determination unit 50 calculates the path distance along the imaginary surface 130 between all two points on the boundary, and determines the maximum value of the path distance as the major axis.

[0061] Figure 12(b) represents the combination of two points with the largest path distance. When the size information determination unit 50 determines that the path distance between point P and point Q on the boundary is the largest, it determines the path distance between point P and point Q as the "major axis" of the imaginary surface 130. When the major axis is determined, the size information determination unit 50 derives the combination of two points with the largest path distance among the paths orthogonal to the major axis.

[0062] Figure 12 (c) represents the combination of two points whose path distance in the direction intersecting the major axis is the largest. When the size information determination unit 50 determines that the path distance between point R and point S on the boundary is the largest, it determines the path distance between point R and point S as the "minor axis" of the imaginary surface 130.

[0063] As described above, the imaginary surface 130 corresponds to the imaginary bottom surface of the lesion 100. The size information determination unit 50 derives the major and minor axes of the bottom surface of the lesion 100 by deriving the major and minor axes of the imaginary surface 130. The display processing unit 46 can display the major and minor axes of the bottom surface of the lesion derived by the imaginary surface derivation unit 48 on the display device 12b. By knowing the major and minor axes of the bottom surface of the lesion, the user can confirm whether the lesion is the size of an object subject to ESD.

[0064] The above describes the basic steps of lesion shape measurement and processing. However, in order to derive the three-dimensional shape information of the hypothetical surface 130 with high accuracy, options for processing in each step can also be prepared for user confirmation.

[0065] Figure 13 This is an example of a confirmation screen displayed on the display device 12b showing the area of ​​interest. In S10, the user operates the user interface 70 to set the area of ​​interest 110, but at this time, the user can also confirm whether the set area of ​​interest 110 is appropriate.

[0066] When the user sets the region of interest 110, the image generation unit 40 generates a three-dimensional image of the biological tissue based on its three-dimensional shape information. The display processing unit 46 combines the three-dimensional image of the biological tissue with an image indicating the location of the region of interest 110 and displays it on the display device 12b. Figure 13 In this process, the region of interest 110 is synthesized at a position separated from the three-dimensional image of the biological tissue along the z-axis direction, but it can also be synthesized at a position that aligns with the z-axis direction. When the outer contour of the region of interest 110 is displayed overlapping the three-dimensional image of the biological tissue with the z-axis-aligned position, the user can verify whether the outer contour of the region of interest 110 aligns with the boundary between the lesion 100 and the biological surface 102. If they do not align, the user can reset the region of interest 110 in a manner that aligns it with the region of the lesion 100.

[0067] In S12, the display processing unit 46 can also combine a three-dimensional image of the biological tissue with an image indicating the position of the reference region 120 and display it on the display device 12b. By combining and displaying the three-dimensional image of the biological tissue with the image indicating the position of the reference region 120, the user can confirm whether the reference region 120 has been set appropriately.

[0068] Furthermore, after the imaginary surface 130 is derived by the imaginary surface derivation unit 48, the display processing unit 46 can also synthesize the imaginary surface 130 on a three-dimensional image of the biological tissue and display it on the display device. Originally, the imaginary surface 130 is obscured by the lesion 100 and cannot be visually recognized, but by synthesizing and displaying the imaginary surface 130 on a three-dimensional image of the biological tissue, the user can confirm whether the imaginary surface 130 has been properly derived.

[0069] exist Figure 5 In S16, the hypothetical surface derivation unit 48 derives the three-dimensional shape information of the hypothetical surface 130 based on the three-dimensional shape information of the reference region 120, which is different from the region of interest 110. However, sometimes there are objects in the reference region 120 that disrupt the continuity of the biological surface. When the hypothetical surface derivation unit 48 performs fitting processing of the three-dimensional shape of the region of interest 110 based on the three-dimensional shape information of the discontinuous biological surface, it cannot estimate the three-dimensional shape information of the hypothetical surface 130 with high accuracy.

[0070] Figure 14 This illustrates an example where a discontinuity 104 is present in the reference region 120. The discontinuity 104 may be a convex or concave portion that impairs the continuity of a biological surface in the reference region 120, or it may be a lesion. During three-dimensional shape fitting, the discontinuity 104 negatively impacts the fitting accuracy; therefore, it is preferable for the user to manually remove it before fitting.

[0071] Figure 15 The user operates the user interface 70 to set the exclusion region 140 in the endoscopic image displayed on the display device 12b. When the operation receiving unit 30 receives a user operation to set the exclusion region 140 in the reference region 120, the imaginary surface derivation unit 48 derives the three-dimensional shape information of the imaginary surface 130 based on the three-dimensional shape information of the reference region 120 after excluding the three-dimensional shape information of the exclusion region 140. As a result, the imaginary surface derivation unit 48 can derive the imaginary surface 130 with high accuracy.

[0072] At this time, the imaginary surface derivation unit 48 can also perform fitting processing on the three-dimensional shape of the excluded region 140 based on the three-dimensional shape information of the reference region 120 after excluding the three-dimensional shape information of the excluded region 140, and derive the correction surface (imaginary surface) of the excluded region 140. In this way, the imaginary surface derivation unit 48 can also derive the imaginary surface 130 of the region of interest 110 based on the corrected three-dimensional shape information of the reference region 120 after correcting the three-dimensional shape information of the excluded region 140.

[0073] The above describes how the information processing device 10b performs the lesion shape measurement function of the processing device 20 when the user generates an examination report after the examination. Alternatively, the information processing device 10a can also cooperate with the image analysis device 8 to perform the lesion shape measurement function of the processing device 20 during endoscopic examination.

[0074] Image analysis device 8 includes image analysis unit 42 and learned model holding unit 60 (reference). Figure 3 The learned model holding unit 60 holds a learned model generated by machine learning using endoscope images for learning and information related to lesion regions contained in the endoscope images as training data. This learned model is configured to detect lesion regions and output the location of the lesion regions when an endoscope image is input.

[0075] During endoscopic examination, the endoscopic observation device 5 displays the endoscopic images captured by the endoscope 7 in real time via the display device 6, and sends these images to the information processing device 10a and the image analysis device 8. In the image analysis device 8, the image analysis unit 42 inputs the acquired endoscopic image into the learned model stored in the learned model holding unit 60. When the endoscopic image is input, the learned model outputs the location information of the lesion area if a lesion area is detected. The image analysis unit 42 sends the lesion area location information output by the learned model along with information identifying the endoscopic image (image ID) to the information processing device 10a.

[0076] In the information processing device 10a, the display processing unit 46 displays the endoscopic image captured by the endoscope 7 on the display device 12a, and displays information indicating the location of the lesion area in the endoscopic image based on the location information of the lesion area and the image ID provided by the image analysis device 8. At this time, the display processing unit 46 synchronizes the displayed endoscopic image with the information indicating the location of the displayed lesion area based on the image ID. For example, Figure 6 As shown, the display processing unit 46 can overlay the boundary line of the lesion 100 on the endoscopic image containing the lesion 100.

[0077] The user operating the information processing device 10a can determine the setting of the region of interest 110 by operating the user interface 70 after confirming that the boundary line of the lesion 100 output by the learned model is correct. At this time, the operation receiving unit 30 accepts the user operation for setting the region of interest on the endoscopic image, and the region of interest setting unit 34 sets the region of interest on the endoscopic image according to the user operation. In this way, when the endoscopic shape measurement system 1 has the function of detecting lesion areas using the learned model, the user can also set the region of interest simply by confirming that the detected lesion area is correct. When the region of interest is set, the information processing device 10a performs... Figure 5 Steps S12, S14, S16, and S18, as shown, determine information related to the size of the imaginary surface 130. This information can be provided to the endoscopic observation device 5, displayed on the display device 6, and communicated to the physician operating the endoscope 7.

[0078] The hypothetical surface derivation unit 48 derives the three-dimensional shape information of the hypothetical surface 130 based on the three-dimensional shape of the surrounding area of ​​the region of interest 110. Therefore, when the surrounding area is small, the three-dimensional data used to estimate the three-dimensional shape of the hypothetical surface 130 is insufficient, making it difficult to derive the three-dimensional shape information of the hypothetical surface 130 with high accuracy. Therefore, the auxiliary information generation unit 44 can generate auxiliary information related to the imaging range of the endoscope 7 during imaging, based on the location of the lesion area provided by the image analysis device 8. For example, if the proportion of the lesion area in the endoscopic image exceeds a predetermined threshold (e.g., 60%), it is impossible to ensure a sufficiently large reference area around the lesion area. Therefore, the auxiliary information generation unit 44 can generate auxiliary information to notify the doctor that the camera of the endoscope 7 is too close to the lesion. The auxiliary information generation unit 44 can provide the generated auxiliary information to the endoscope observation device 5, which displays the auxiliary information on the display device 6, allowing the doctor operating the endoscope 7 to confirm it. Thus, the doctor can identify situations where the camera of the endoscope 7 is too close to the lesion and move the endoscope 7 away from the lesion. Therefore, by ensuring a sufficiently large peripheral area around the lesion, the imaginary surface derivation unit 48 can derive the three-dimensional shape information of the imaginary surface 130 with high accuracy.

[0079] The present invention has been described above based on several embodiments. These embodiments are illustrative, and those skilled in the art will understand that various modifications can be made to the combination of these constituent elements and processing procedures, and such modifications are also within the scope of the present invention.

[0080] Industrial availability

[0081] This invention can be applied to the field of measuring the shape of lesion sites.

[0082] Label Explanation

[0083] 1... Endoscopic shape measurement system, 5... Endoscopic observation device, 6... Display device, 7... Endoscope, 8... Image analysis device, 9... Image storage server, 10a, 10b... Information processing device, 12a, 12b... Display device, 20... Processing device, 30... Operation receiving unit, 32... Region setting unit, 34... Region of interest setting unit, 36... Reference region setting unit, 38... Three-dimensional information acquisition unit, 40... Image generation unit, 42... Image analysis unit, 44... Auxiliary information generation unit, 46... Display processing unit, 48... Imaginary surface derivation unit, 50... Size information determination unit, 60... Learned model retention unit, 70... User interface.

Claims

1. A shape measurement system for an endoscope, comprising a processor, characterized in that, The processor Endoscopic images of biological tissues within a body cavity, captured by an endoscope, are displayed on a display device. Accepts user input to define a region of interest in an endoscopic image, and defines the region of interest based on the user input. Obtain three-dimensional shape information of biological tissues captured by endoscopy. Based on the three-dimensional shape information of a region different from the region of interest, the three-dimensional shape information of an imaginary surface in the region of interest is derived, wherein the imaginary surface is the obscured biological surface contained in the endoscopic image. Based on the derived three-dimensional shape information of the imaginary surface, determine the information related to the size of the imaginary surface.

2. The endoscopic shape measurement system according to claim 1, characterized in that, The processor Define a reference region that surrounds the area of ​​interest. The three-dimensional shape information of the imaginary surface is derived based on the three-dimensional shape information of the reference area.

3. The endoscopic shape measurement system according to claim 2, characterized in that, The processor accepts user operations for setting reference regions in endoscopic images. Set a reference area in the endoscopic image based on user input.

4. The endoscopic shape measurement system according to claim 1, characterized in that, The processor Generate a 3D image of the biological tissue based on its 3D shape information. The three-dimensional image of the biological tissue and the image representing the location of the region of interest are combined and displayed on the display device.

5. The endoscopic shape measurement system according to claim 2, characterized in that, The processor Generate 3D images of biological tissues based on their 3D shape information. The three-dimensional image of the biological tissue and the image representing the location of the reference area are combined and displayed on the display device.

6. The endoscopic shape measurement system according to claim 2, characterized in that, The processor Accept user operations for setting exclusion zones in the reference area. Based on the three-dimensional shape information of the reference region after excluding the excluded region, the three-dimensional shape information of the imaginary surface is derived.

7. The endoscopic shape measurement system according to claim 4, characterized in that, The processor synthesizes a hypothetical surface in a three-dimensional image of a biological tissue and displays it on a display device.

8. The endoscopic shape measurement system according to claim 1, characterized in that, The processor performs 3D shape fitting processing based on the 3D shape information of a region different from the region of interest, thereby deriving the 3D shape information of the imaginary surface in the region of interest.

9. The endoscopic shape measurement system according to claim 1, characterized in that, The endoscopic shape measurement system also has a learned model, which is generated by machine learning using endoscopic images and information related to lesion areas contained in the endoscopic images as training data. When an endoscopic image is input, the learned model outputs the location of the lesion area. The processor displays information representing the location of the lesion area, output by the learned model.

10. The endoscopic shape measurement system according to claim 1, characterized in that, The endoscopic shape measurement system also includes a learned model, which is generated through machine learning by using endoscopic images and information related to lesion areas contained in the endoscopic images as training data. When an endoscopic image is input, the learned model outputs the location of the lesion area. The processor generates auxiliary information related to the imaging range of the endoscope during imaging based on the location of the lesion area output by the learned model, and displays the auxiliary information on the display device.

11. The endoscopic shape measurement system according to claim 2, characterized in that, The processor sets a reference region based on the region of interest.

12. A shape measurement method for an endoscope, wherein, Endoscopic images of biological tissues within a body cavity, captured by an endoscope, are displayed on a display device. Accepts user actions for defining regions of interest in endoscopic images. Based on user input, the region of interest is defined within the endoscopic image. Obtain three-dimensional shape information of biological tissues captured by endoscopy. Based on the three-dimensional shape information of a region different from the region of interest, the three-dimensional shape information of an imaginary surface in the region of interest is derived, wherein the imaginary surface is the obscured biological surface contained in the endoscopic image. Based on the derived three-dimensional shape information of the imaginary surface, determine the information related to the size of the imaginary surface.

Images