Hollow organ three-dimensional model construction method, model, electronic equipment and storage medium

Abstract

The invention discloses a hollow organ three-dimensional model construction method, a model, electronic equipment and a storage medium. The method comprises the following steps: acquiring a two-dimensional plane graph by using digestive endoscopy to determine the contour of a target hollow organ in an expanded state, performing three-dimensional modeling by using a CT image to obtain an orthogonal projection drawing, obtaining the contour of a target hollow organ in a non-expanded state by combining the contour of the target hollow organ in the expanded state, and calculating an expansion coefficient through perimeter, expanding the hollow viscera in the non-expanded state according to the expansion coefficient to obtain an expanded target hollow viscera three-dimensional model; the digestive endoscopy image and the CT image are both original examination habits and techniques, the enlargement condition of the gastral cavity after inflation can be known under the condition that the original examination habits and techniques are not separated, and perception of the relation between the gastral cavity and adjacent tissue is recovered.

Description

Technical Field

[0001] The present invention relates to the technical field of the construction of hollow organ expansion models, and specifically, to a method for constructing a three-dimensional model of a hollow organ, a model, an electronic device, and a storage medium. Background Art

[0002] In the related art, an endoscope can only visually observe the situation inside the digestive tract lumen, and cannot obtain the field of view outside the lumen at the corresponding moment, nor can it take CT images simultaneously during use; CT images can show three-dimensional images, but when taking CT images, the patient is usually in a natural state, that is, when the gastric cavity is not inflated or filled with water. During endoscopy, the gastric cavity needs to be continuously inflated to stretch the mucosa to obtain the best observation field of view. This results in a closer relationship between the inflated gastric cavity and the surrounding adjacent tissues due to expansion, while the previously taken CT images do not consider this factor, thus affecting the doctor's endoscopic diagnosis and treatment.

[0003] Therefore, when the inflated and expanded conditions of hollow organs such as the gastric cavity and the surrounding adjacent tissues being compressed cannot be seen, correct information cannot be provided to guide the doctor's understanding of the real situation. Summary of the Invention

[0004] The purpose of the present invention is to solve the above problems, and provide a method for constructing a three-dimensional model of a hollow organ, a model, an electronic device, and a storage medium, so as to solve the problem that when the inflated conditions of hollow organs such as the gastric cavity and the surrounding adjacent tissues being compressed cannot be seen, correct information cannot be obtained, thereby guiding the doctor's understanding of the real situation.

[0005] To solve the above problems, the present invention provides the following technical solutions: On the one hand, a method for constructing a three-dimensional model of an expanded hollow organ includes Determining its contour according to the two-dimensional plane diagram of the target hollow organ in the expanded state; Constructing a three-dimensional model based on the image of the target hollow organ in the unexpanded state, and determining the orthographic projection diagram of the target hollow organ in the unexpanded state according to the three-dimensional model; Determining all contour points of the target hollow organ in the expanded and unexpanded states according to the two-dimensional plane diagram and the orthographic projection diagram; determining the contour diagrams of the target hollow organ in the expanded and unexpanded states according to the angles between the contour points and the coordinate origin; Calculating the perimeters of the target hollow organ in the expanded state and the target hollow organ in the unexpanded state respectively, and calculating the expansion coefficient according to the perimeters of the two; Obtaining the three-dimensional model of the target hollow organ after expansion according to the expansion coefficient and the three-dimensional model in the unexpanded state.

[0006] In related embodiments, determining the orthogonal projection diagram of the target hollow organ in the unexpanded state based on the three-dimensional model includes: Load the three-dimensional model, ignore the y coordinate of each point, obtain the values of its x and z axis coordinates, and draw a planar diagram of the target hollow organ according to the x and z values of each point to obtain the orthogonal projection diagram of the three-dimensional model on the XZ axis.

[0007] In related embodiments, determining all the contour points of the expanded and unexpanded states of the target hollow organ based on the two-dimensional planar diagram and the orthogonal projection diagram includes: From the top to the bottom of the two diagrams of the two-dimensional planar diagram and the orthogonal projection diagram respectively, draw a straight line parallel to the x-axis at a set distance along the z-axis direction. This straight line will pass through the orthogonal projection and the two-dimensional planar diagram respectively and generate two intersection points with them. These two intersection points are the points with the minimum and maximum x values. Draw lines and take points in turn until all the contour points of the target hollow organ in the expanded and unexpanded states are obtained.

[0008] In related embodiments, determining the contour diagrams of the target hollow organ in the expanded and unexpanded states based on the angles between the contour points and the origin of coordinates includes: Calculate the angle between the line connecting the contour point and the origin of coordinates and the x or z axis. By analogy, calculate the angle sizes of all the contour points and sort them in ascending order to obtain the ordered contour points in the clockwise direction.

[0009] In related embodiments, calculating the perimeter of the target hollow organ in the unexpanded state is the sum of the Euclidean distances between all adjacent two contour points in the contour diagram of the target hollow organ in the unexpanded state.

[0010] In related embodiments, calculating the perimeter of the target hollow organ in the expanded state includes: The ratio of the length of the real object existing in the two-dimensional planar diagram to the pixel distance at the same position corresponding to the length of the real object in the two-dimensional planar diagram is the relationship coefficient; Calculate the sum of the Euclidean distances between all adjacent two contour points according to the contour of the target hollow organ in the expanded state as the pixel perimeter; The perimeter of the target hollow organ in the expanded state is the product of the pixel perimeter and the relationship coefficient.

[0011] In related embodiments, obtaining the three-dimensional model of the target hollow organ after expansion based on the expansion coefficient and the three-dimensional model in the unexpanded state includes: Define the scaling ratio through instructions to enlarge the three-dimensional model according to the expansion coefficient; Translate the coordinates of each vertex of the enlarged three-dimensional model grid relative to the centroid so that the centroid moves to the origin of coordinates; Scale each vertex of the enlarged three-dimensional model; Move the magnified grid back to its original centroid position; Update the vertices of the magnified 3D model grid.

[0012] In a second aspect, an enlarged three-dimensional model of a hollow organ, which is a model constructed based on a method for constructing an enlarged three-dimensional model of a hollow organ.

[0013] In a third aspect, an electronic device includes a memory and a processor. Computer-readable instructions are stored in the memory. When the computer-readable instructions are executed by the processor, the processor executes a method for constructing an enlarged three-dimensional model of a hollow organ.

[0014] In a fourth aspect, a computer-readable storage medium stores a computer program. When the computer program runs on a computer, the computer executes a method for constructing an enlarged three-dimensional model of a hollow organ.

[0015] Compared with the prior art, the present invention has the following beneficial effects: The present invention uses a digestive endoscopy examination to obtain a two-dimensional plan view to determine the outline of the target hollow organ in the enlarged state, uses CT image three-dimensional modeling to obtain an orthogonal projection view, combines the outline of the target hollow organ in the enlarged state to obtain the outline diagrams of the target hollow organ in both the enlarged and unenlarged states, calculates the enlargement coefficient through the perimeter ratio of the two, and enlarges the hollow organ in the unenlarged state according to this enlargement coefficient to obtain an enlarged three-dimensional model of the target hollow organ; both the digestive endoscopy image and the CT image are the original examination habits and techniques, and the enlargement of the gastric cavity after inflation can be understood without departing from the original examination habits and techniques, and the perception of the relationship between the gastric cavity and adjacent tissues can be restored. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following will briefly introduce the drawings required for the description of the specific embodiments or the prior art. Obviously, the following drawings are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts, where: Figure 1 It is a plan view of the gastric cavity after inflation taken by X-ray in ERCP.

[0017] Figure 2 It is an orthogonal projection view of the gastric cavity without inflation based on the three-dimensional reconstruction model.

[0018] Figure 3 It is to extract the complete contour points of the inflated gastric cavity (large) and the non-inflated gastric cavity (small).

[0019] Figure 4To measure the pixel diameter of the endoscope lumen.

[0020] Figure 5 It is a comparison diagram before and after expanding the three-dimensional reconstruction model (non-inflated gastric cavity) according to a coefficient.

[0021] Figure 6 It is a flowchart of a method for constructing a three-dimensional model of an enlarged hollow organ. Specific embodiments

[0022] To make the objectives, technical solutions and advantages of the present invention clearer, the following will be combined with Figures 1 to 6 The present invention will be further described in detail. The described embodiments should not be regarded as limitations of the present invention. All other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of protection of the present invention.

[0023] This embodiment is used to solve the problems in upper gastrointestinal endoscopy diagnosis and treatment, such as the expansion of the gastric cavity caused by inflation during endoscopy, which changes the anatomical relationship with the surrounding adjacent tissues, thus affecting the doctor's surgical judgment and operation risk. A method for constructing a three-dimensional model of an enlarged hollow organ is proposed. As Figure 6 shown, the specific steps are as follows: Considering that currently, endoscopic retrograde cholangiopancreatography (ERCP) also performs relevant operations through the oral route of the endoscope through the gastric cavity and finally reaches the duodenum. At the same time, X-ray fluoroscopy images are taken for observation. With the help of this operation and related equipment, when passing through the gastric cavity, simulate a conventional gastroscopy to inflate the gastric cavity, and adjust the X-ray camera to take pictures with the complete contour of the inflated gastric cavity, which are recorded as post-inflation pictures, as Figure 1 shown; 2) Use the CT imaging data of the same patient, and with the help of the totalsegmentator software, perform three-dimensional reconstruction on the patient's thoracic and abdominal organs, and separately extract the part of the gastric cavity; 3) Use the existing open3d library to load each point in the three-dimensional model, ignore the y coordinate of each point, obtain the values of its x and z axis coordinates, and then use the matplotlib library to draw a planar graph of the stomach according to the x and z values of each point, so as to obtain the orthogonal projection of the three-dimensional model on the XZ axis, and get a two-dimensional planar graph with the same viewing angle as the gastric cavity picture taken by ERCP, which is recorded as the pre-inflation picture, as Figure 2 shown; Since the reconstructed three-dimensional model may be tilted or inverted, obtaining its coordinate values in the vertical and horizontal directions (Z axis and X axis) helps to ensure that the placement position of the three-dimensional model is consistent with the vertical direction of the human body, and the extracted coordinate information is helpful for the progress of the following steps.

[0024] 4) Use the PAIR software to outline the entire gastric cavity in the X-ray images during ERCP and save it as a TAR compressed file containing coordinate information; decompress the compressed package to obtain a pose.json file, which contains the pixel coordinates of each contour point.

[0025] 5) Obtain all the contour points of the gastric cavity in the two images: It can be imagined that the projection is a plane composed of many points. Therefore, draw a straight line parallel to the x-axis every 1 mm along the z-axis direction from the top to the bottom of the two images respectively. This straight line will pass through the projection plane and generate two intersection points with it. These two intersection points are the points with the smallest and largest x values. Draw lines and take points in sequence until the bottom of the image to obtain all the contour points of the entire gastric cavity; if directly extracting the contour points of one image, all the extracted points have no coordinate information and are a bunch of scattered points. Moreover, there are many points on one image, and it is uncertain whether all the extracted points are contour points. Drawing lines and taking points in sequence can ensure that the extracted points are all contour points and have coordinate information such as (1, y), (2, y).

[0026] 6) Sort the extracted contour points in the clockwise direction: Since when extracting the contour points, they are not taken step by step in the clockwise or counterclockwise direction, the perimeter cannot be calculated. Therefore, the contour points must be sorted in the clockwise or counterclockwise direction first. The specific method is: use the to calculate the angle between each point and the origin of coordinates through the x and z-axis coordinates of two points, and calculate the angle sizes of all contour points in turn and sort them in ascending order to obtain the ordered contour points in the clockwise direction, as shown in Figure 3 ; after drawing lines and taking points, a bunch of points with coordinate information (1, y), (2, y), (2, y1), (3, y2) will be obtained. The computer needs a rule to process these points. This rule is to calculate the angle sizes and let it arrange these points in sequence, which is the contour of the image.

[0027] 7) Calculate the perimeter of the gastric cavity contour: ① Calculate the perimeter of the gastric cavity in the non-inflated image: Calculate the Euclidean distance between every two points among all the contour points, and add up the distances between all points to obtain the approximate perimeter of the entire contour. The specific calculation method is as follows: The coordinates of the ordered point 1 are: ; The coordinates of the ordered point 2 are: ; The Euclidean distance between the two is:

[0028] ② Calculate the perimeter of the gastric cavity in the inflated picture: It should be noted that for the gastric model reconstructed three-dimensionally based on CT images, the orthogonal projection extracted, that is, the contour point coordinates of the non-inflated picture, are actual coordinates, while the gastric cavity plane diagram taken by X-ray outlined by the PAIR software, that is, the picture after inflation, outputs the pixel coordinates of the contour rather than the actual coordinates of the inflated gastric cavity. Therefore, the sum of the Euclidean distances calculated through pixel coordinates is not the actual perimeter of the inflated gastric cavity, and the two need to be converted. The specific method is as follows: a: First, calculate the relationship between the pixel distance and the actual distance of the gastric cavity contour extracted from the picture after inflation: The known true diameter of the endoscope used during ERCP surgery is 13.5 mm, and the resolution of the inflated image taken by X-ray is 2840 * 2874. Load this picture into the computer paint software, and use the drawing tool to draw a line segment perpendicular to the endoscope tube body (i.e., measure the diameter, as Figure 4 ), and the pixel coordinates at both ends of the endoscope lumen diameter can be obtained. Subsequently, calculate the pixel distance between the two pixels. Through the ratio of the true diameter of the endoscope lumen to the pixel diameter on the picture: actual physical distance / pixel distance = relationship coefficient R.

[0029] b: Calculate the perimeter of the inflated gastric contour: Calculate the Euclidean distance between every two points among all the contour points, and add up the distances between all the points to obtain the approximate perimeter of the entire contour. The specific calculation method is the same as that for calculating the non-inflated contour diagram, but note that the perimeter obtained at this time is the pixel distance.

[0030] c: Multiply this pixel distance by the relationship coefficient R to obtain the actual perimeter of the inflated gastric cavity: For example, if the actual distance of one pixel is 0.147 mm, and the calculated perimeter is 5238.252 pixels, then its actual perimeter is: 5238.252 * 0.147 = 770.023 mm.

[0031] 8) Calculate the gastric cavity expansion coefficients of the two pictures of the non-inflated and inflated ones according to the perimeter: For example, if the perimeter of the non-inflated gastric cavity is 582.687 mm and the perimeter of the inflated gastric cavity is 770.023 mm, the calculated magnification coefficient is 770.023 mm / 582.687 mm = 1.321 9) Scale up the reconstructed three-dimensional gastric cavity model by this ratio, as Figure 5 shown, so as to guide the doctor to correctly perceive the anatomical relationship of the tissues adjacent to the inflated gastric cavity with the help of this corrected model to assist in relevant diagnoses and operations. The specific operations are as follows: Define the scaling ratio through instructions (scale_factor = 1.321 # scale up the non-inflated three-dimensional model reconstructed based on CT images by 1.321 times) Translate the coordinates of each vertex of the mesh relative to the centroid so that the centroid is moved to the origin of coordinates (vertices_centered = vertices – centroid) Scale each vertex (scaled_vertices = vertices_centered * scale_factor) Move the enlarged mesh back to the original centroid position (scaled_vertices += centroid) Update the vertices of the mesh (mesh.vertices = o3d.utility.Vector3dVector(scaled_vertices)) Through testing and verification of a considerable number of pictures, a general expansion coefficient after gastric cavity inflation can be obtained to fill the current knowledge gap. The calculation method of this gastric cavity expansion coefficient can refer to solving the clinical impact caused by deformation of other hollow organs such as the bladder and uterus.

[0032] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor; when the processor executes the computer program, it implements the steps of a method for constructing a three-dimensional model of an enlarged hollow organ.

[0033] The electronic device can be a desktop computer, a notebook, a palm computer, a cloud server and other electronic devices. The electronic device can include but is not limited to a processor and a memory. Those skilled in the art can understand that the figure is only an example of the electronic device and does not constitute a limitation on the electronic device, and may include more or fewer components than shown in the figure, or different components.

[0034] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0035] The memory can be an internal storage unit of an electronic device, for example, the hard disk or memory of the electronic device. The memory can also be an external storage device of the electronic device, for example, a plug-in hard disk equipped on the electronic device, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. The memory can also include both the internal storage unit of the electronic device and the external storage device. The memory is used to store computer programs and other programs and data required by the electronic device.

[0036] In several embodiments provided by the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architectures, functions, and operations of devices, methods, and computer program products according to multiple embodiments of the present invention. In this regard, each block in the flowchart or block diagram can represent a module, a program segment, or a part of code, and the part of the module, program segment, or code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than marked in the accompanying drawings. For example, two consecutive blocks can actually be executed substantially in parallel, and they can sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and the combination of blocks in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system that performs the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

[0037] In addition, each functional module in various embodiments of the present invention can be integrated together to form an independent part, or each module can exist alone, or two or more modules can be integrated to form an independent part.

[0038] If the above-described functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. It should be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements not only includes those elements, but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the existence of additional identical elements in the process, method, article or device comprising the said element.

[0039] The above are only the preferred embodiments of the present invention and are not used to limit the present invention. For those skilled in the art, the present invention can have various changes and modifications. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention. It should be noted that similar reference numerals and letters indicate similar items in the following drawings. Therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings.

[0040] The above is only the specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention can easily think of changes or replacements, which should all be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention should be subject to the protection scope of the claims.

Claims

1. A method for constructing an enlarged three-dimensional model of a hollow organ, characterized in that: include Determine the contour of the target hollow organ according to the two-dimensional plane image in the enlarged state; constructing a three-dimensional model based on the image of the target hollow organ in a non-enlarged state, and determining an orthogonal projection of the target hollow organ in a non-enlarged state based on the three-dimensional model; According to the two-dimensional plane diagram and the orthogonal projection diagram, all contour points of the target hollow organ in the expanded and non-expanded states are determined; according to the angle between the contour points and the coordinate origin, the contour diagram of the target hollow organ in the expanded and non-expanded states is determined; Calculate the perimeters of the target hollow organ in an enlarged state and the target hollow organ in a non-enlarged state respectively, and calculate the enlargement coefficient according to the perimeters of the two; The enlarged three-dimensional model of the target hollow organ is obtained according to the enlargement coefficient and the three-dimensional model in the non-enlarged state.

2. The method for constructing an enlarged three-dimensional model of a hollow organ according to claim 1, characterized in that: The orthogonal projections of the target hollow organ in the unenlarged state determined based on the three-dimensional model include: When loading the three-dimensional model, the y coordinate of each point is ignored, and the values ​​of its x and z axis coordinates are obtained. A plan view of the target hollow organ is drawn according to the x and z values ​​of each point, thereby obtaining an orthogonal projection view of the three-dimensional model on the XZ axis.

3. The method for constructing an enlarged three-dimensional model of a hollow organ according to claim 1, characterized in that: Based on the two-dimensional plan view and orthogonal projection view, all contour points of the target hollow organ in the enlarged and non-enlarged states are determined, including: Draw a straight line parallel to the x-axis at a set distance along the z-axis direction from the top to the bottom of the two-dimensional plane image and the orthogonal projection image. This straight line will pass through the orthogonal projection and the two-dimensional plane image respectively, and produce two intersection points with them. These two intersection points are the points with the minimum and maximum x values. Draw lines and take points in sequence to obtain all the contour points of the target hollow organs in both expanded and non-expanded states.

4. The method for constructing an enlarged three-dimensional model of a hollow organ according to claim 1, characterized in that: According to the angle between the contour point and the coordinate origin, the contour map of the target hollow organ in the expanded and unexpanded states is determined, including: Calculate the angle between the line connecting the contour point and the origin of the coordinate system and the x or z axis. Similarly, calculate the angles of all contour points and sort them in ascending order to obtain the contour points in a clockwise direction.

5. The method for constructing an enlarged three-dimensional model of a hollow organ according to claim 1, characterized in that: The perimeter of the target hollow organ in the unenlarged state is calculated as the sum of the Euclidean distances between all two adjacent contour points in the contour map of the target hollow organ in the unenlarged state.

6. The method for constructing an enlarged three-dimensional model of a hollow organ according to claim 1, characterized in that: Calculation of the circumference of the target hollow organ in the enlarged state includes: The ratio of the length of a real object existing in the two-dimensional plane image to the pixel distance at the same position in the two-dimensional plane image corresponding to the length of the real object is the relationship coefficient; According to the contour of the target hollow organ in the enlarged state, the sum of the Euclidean distances between all two adjacent contour points is calculated as the pixel perimeter; The perimeter of the target hollow organ in the enlarged state is the product of the pixel perimeter and the relationship coefficient.

7. The method for constructing an enlarged three-dimensional model of a hollow organ according to claim 1, characterized in that: The three-dimensional model of the target hollow organ after enlargement obtained according to the enlargement coefficient and the three-dimensional model in the non-enlarged state includes: Define the scaling ratio through the command to enlarge the 3D model according to the expansion factor; The coordinates of each vertex of the enlarged three-dimensional model mesh are translated relative to the center of mass, so that the center of mass moves to the origin of the coordinates; Scaling each vertex of the enlarged three-dimensional model; Move the enlarged grid back to its original centroid position; Update the vertices of the enlarged 3D model mesh.

8. An enlarged three-dimensional model of a hollow organ, characterized in that: A model constructed based on the method for constructing an enlarged hollow organ three-dimensional model according to any one of claims 1 to 7.

9. An electronic device, comprising a memory and a processor, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the electronic device comprises: The processor is enabled to execute the method for constructing an enlarged three-dimensional model of a hollow organ according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The readable storage medium stores a computer program, and when the computer program is executed on a computer, the computer is enabled to execute the method for constructing an enlarged three-dimensional model of a hollow organ according to any one of claims 1 to 7.

Images