A medical image processing method based on three-dimensional visualization

Abstract

The present invention discloses a medical image processing method based on three-dimensional visualization, including: obtaining the correlation between the medical two-dimensional image and the three-dimensional image based on the matching loss of key points of the medical two-dimensional image for prior; designing a differentiable rendering channel by using the correlation; simulating the influence of triangles on the image plane in combination with a probability map, and using an aggregation function to merge color mappings to obtain a rendering output based on the probability map and relative depth. Through a special technique for three-dimensional mesh reconstruction of medical two-dimensional images, that is, designing a differentiable rendering channel, the present invention increases the rendering output and shape fitting, quickly, accurately, clearly, and directly helps the efficient three-dimensional visualization conversion of the input image in the computer, and greatly helps medical staff to accurately judge colon diseases.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a medical image processing method based on three-dimensional visualization. Background Art

[0002] The human body three-dimensional reconstruction technology is mainly characterized by medicine and information engineering, and reconstructs the three-dimensional anatomical structure of human tissues in a virtual reality environment. The medical images obtained in modern medical research and clinical diagnosis and treatment are diverse, such as CT images, X-rays, nuclear medicine images, magnetic resonance (MRI) images, ultrasound images, and various electronic endoscope images, etc. However, these images provide two-dimensional images without processing; in the face of two-dimensional images, clinicians can only estimate the size and shape of the lesions based on experience and conceive the three-dimensional geometric relationship between the lesions and their surrounding tissues, which brings great difficulties to the accuracy and convenience of diagnosis and treatment.

[0003] The three-dimensional visualization technology of medical images is to visually display the information data obtained by digital imaging devices such as MRI and CT on a computer in a three-dimensional effect, so as to provide structural information that cannot be obtained by traditional means; after reconstructing a three-dimensional image with a computer and vividly displaying a three-dimensional image of a human organ on the screen, clinicians can conveniently perform operations such as translation, rotation, and dissection on the reconstructed image, and can also improve the preoperative evaluation and even perform a visual simulation surgery. In this way, medical staff can more fully understand the nature of the disease and the three-dimensional structural relationship of its surrounding tissues, so as to help medical staff make an accurate diagnosis and formulate a correct surgical plan, and ultimately achieve the purpose of improving the accuracy and scientific nature of diagnosis and treatment.

[0004] The existing virtual endoscopy imaging technique (CTVE) combines surface shaded display and volume rendering method on the basis of volume data to simulate a three-dimensional space environment, and performs computer data post-processing in the cavity of the organ to be examined to display an image, which is similar to the effect of a fiber endoscope and is commonly used in the cavity organs such as the larynx, bronchus, colon, biliary tract, and stomach; however, what CTVE observes is only the image of the lesion, lacking tissue specificity and unable to perform a biopsy; secondly, the detection sensitivity for flat lesions is relatively low; in addition, CTVE cannot observe the true color change and details of the lumen intima, and it is impossible to distinguish the residual feces in the colon from polyps and masses, and insufficient inflation of the intestinal cavity also causes difficulty in observation; moreover, it is impossible to effectively reconstruct a three-dimensional image by computer, that is, there are technical defects in the three-dimensional visualization processing process, which is quite a headache for medical staff. Summary of the Invention

[0005] The purpose of this section is to outline some aspects of the embodiments of the present invention and briefly introduce some preferred embodiments. Simplifications or omissions may be made in this section, as well as in the abstract and title of the present application, to avoid obscuring the purpose of this section, the abstract, and the title. However, such simplifications or omissions shall not be used to limit the scope of the present invention.

[0006] In view of the above existing problems, the present invention is proposed.

[0007] To solve the above technical problems, the present invention provides the following technical solutions: Based on the matching loss of key points of medical two-dimensional images for prior knowledge, the correlation between the medical two-dimensional images and three-dimensional images is obtained; a differentiable rendering channel is designed using the correlation; the influence of triangles on the image plane is simulated by combining probability maps, and an aggregation function is used to merge color mappings to obtain a rendering output based on the probability maps and relative depths.

[0008] As a preferred embodiment of the medical image processing method based on three-dimensional visualization according to the present invention, wherein: the medical two-dimensional images are rendered and output through the designed differentiable rendering channel to construct a three-dimensional mesh, and further includes

[0009] Reconstructing a single-view mesh and image-based shape fitting.

[0010] As a preferred embodiment of the medical image processing method based on three-dimensional visualization according to the present invention, wherein: the reconstruction of the single-view mesh includes

[0011] Given an input image, a shape and color generator generates a triangular mesh and its corresponding color;

[0012] Inputting it into a soft rasterizer;

[0013] The soft ray layer simultaneously renders the contour and color image and provides a rendering-based error signal by comparing with the ground truth.

[0014] As a preferred embodiment of the medical image processing method based on three-dimensional visualization according to the present invention, wherein: the image-based shape fitting includes

[0015]

[0016] Wherein, R(M(p,θ,t)) is the rendering function for the mesh M to generate a rendered image, which is parameterized by the pose θ, translation t, and non-rigid deformation parameter p, and I is the rendering output.

[0017] As a preferred embodiment of the medical image processing method based on three-dimensional visualization according to the present invention, wherein: designing a differentiable rendering channel using the correlation includes defining external variables for the environmental settings and internal attributes describing the specific properties of the model;

[0018] The external variables include a camera and lighting conditions;

[0019] The internal attributes include a triangular mesh and vertex appearance.

[0020] As a preferred solution of the medical image processing method based on three-dimensional visualization according to the present invention, wherein: the vertex appearance includes color and material.

[0021] As a preferred solution of the medical image processing method based on three-dimensional visualization according to the present invention, wherein: the combining probability map to simulate the influence of a triangle on the image plane includes,

[0022]

[0023] where σ is a positive scalar that controls the sharpness of the probability distribution, and δ i j is a sign indicator, δ i j = {+1, if Pi ∈ fi; -1, otherwise}, and d(i,j) is the shortest distance from Pi to the edge of triangle fj.

[0024] As a preferred solution of the medical image processing method based on three-dimensional visualization according to the present invention, wherein: the aggregation function includes,

[0025]

[0026] where C b is the background color, the weight w j = w b = 1, C j is the color mapping, I is the rendering output, and A S ({C j ) is the aggregation function.

[0027] As a preferred solution of the medical image processing method based on three-dimensional visualization according to the present invention, wherein: it further includes,

[0028]

[0029] where z i j represents the normalized inverse depth of the 3D point on triangle fi, and its 2D projection is Pi; ε is a small constant to enable the background color, and γ controls the sharpness of the aggregation function.

[0030] Advantages of the present invention: Through a special technique for three-dimensional grid reconstruction of medical two-dimensional images, that is, designing a differentiable rendering channel, the present invention increases the rendering output and shape fitting, and quickly, accurately, clearly, and directly helps the efficient three-dimensional visualization conversion of the input image in the computer, greatly assisting medical staff in accurately judging colon diseases. Description of the Drawings

[0031] Figure 1 It is a schematic flowchart of a medical image processing method based on three-dimensional visualization according to an embodiment of the present invention;

[0032] Figure 2 It is a probability graph of triangles representing different positive scalars using Euclidean distance in a medical image processing method based on three-dimensional visualization according to an embodiment of the present invention;

[0033] Figure 3 It is a schematic diagram of a single-view grid reconstruction framework in a medical image processing method based on three-dimensional visualization according to an embodiment of the present invention;

[0034] Figure 4 It is a schematic diagram of three-dimensional linear interpolation in a medical image processing method based on three-dimensional visualization according to an embodiment of the present invention. Detailed Embodiments

[0035] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0036] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily need to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein.

[0037] It should be understood that in various embodiments of the present invention, the magnitude of the serial numbers of the various processes does not mean the order of execution, and the order of execution of the various processes should be determined by their functions and internal logics, and should not constitute any limitation to the implementation process of the embodiments of the present invention.

[0038] It should be understood that in the present invention, "including" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units need not be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products, or devices.

[0039] It should be understood that in the present invention, "a plurality of" means two or more. "And / or" is merely a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone. The character " / " generally indicates that the associated objects before and after are in an "or" relationship. "Including A, B, and C" and "including A, B, C" mean that all of A, B, and C are included. "Including A, B, or C" means including any one of A, B, and C. "Including A, B, and / or C" means including any one, any two, or all three of A, B, and C.

[0040] It should be understood that in the present invention, "B corresponding to A", "B corresponding to A relatively", "A corresponding to B relatively", or "B corresponding to A relatively" means that B is associated with A, and B can be determined according to A. Determining B according to A does not mean determining B only according to A. B can also be determined according to A and / or other information. The matching of A and B means that the similarity between A and B is greater than or equal to a preset threshold.

[0041] Depending on the context, as used herein, "if" can be interpreted as "when", "while", "in response to determining", or "in response to detecting".

[0042] The technical solutions of the present invention will be described in detail below with specific embodiments. These specific embodiments below can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.

[0043] Embodiment 1

[0044] Although current imaging devices such as CT, MRI, PET, and ultrasound can collect three-dimensional data, traditional two-dimensional displays can only present two-dimensional planar information. Doctors need to adjust the viewing angle to obtain more angular scale information.

[0045] Presenting three-dimensional information to doctors directly in a stereoscopic display form during surgery can enable doctors to obtain more comprehensive and accurate information on the structural morphology of the patient's organs. However, due to the current development dilemma of three-dimensional visualization technology, it is still not possible to well convert two-dimensional images into three-dimensional images through a computer for medical staff to provide auxiliary diagnosis. Especially for patients with colon diseases, clear three-dimensional visualization technology can greatly help medical staff in treating them.

[0046] Refer to Figures 1 to 4 , which is an embodiment of the present invention, and provides a medical image processing method based on three-dimensional visualization, specifically including the following steps:

[0047] S1: Obtain the correlation between the medical two-dimensional image and the three-dimensional image based on the matching loss of the key points of the medical two-dimensional image. It should be noted that the prior includes:

[0048] Input a one-dimensional one-hot encoded conditional vector;

[0049] Map this vector to a one-dimensional vector of length (channel length * channel width) through a linear layer;

[0050] Reshape this one-dimensional vector into a two-dimensional matrix with the same channel length and width;

[0051] Superimpose this matrix as a new channel on the three-dimensional matrix input to the neural network.

[0052] Furthermore, the superimposing calculation process includes:

[0053] (1) Set the maximum number of iterations;

[0054] (2) Stop the iteration when the mean square error is less than the threshold;

[0055] (3) Stop the iteration when the difference between two change matrices is less than the threshold;

[0056] (4) Set the maximum distance between corresponding point pairs and use it in (1).

[0057] S2: Design a differentiable rendering channel using the correlation. It should be noted in this step that:

[0058] Define external variables for environmental settings and internal attributes for describing model-specific properties;

[0059] External variables include the camera and lighting conditions;

[0060] Internal attributes include the triangular mesh and vertex appearance;

[0061] Vertex appearance includes color and material.

[0062] Refer to Figure 4 , and obtain the required three-dimensional coordinates by linearly interpolating the values of other points in the cube through the given vertex values, as follows:

[0063] x d =(x - x 0 ) / (x 1 - x 0 )

[0064] y d = (y - y 0 ) / (y 1 - y 0 )

[0065] z d = (Z - Z 0 ) / (z 1 - z 0 )

[0066] c = c 000 (1 - x d )(1 - y d )(1 - z d ) +

[0067] c 100 x d (1 - y d )(1 - z d ) +

[0068] c 010 (1 - x d )y d (1 - z d ) +

[0069] c 001 (1 - x d )(1 - y d )z d +

[0070] c 101 x d (1 - y d )z d +

[0071] c 011 (1 - x d )y d z d +

[0072] c 110 x d y d (1 - z d ) +

[0073] c 111 x d y d z d

[0074] Where the coordinates (x, y, z) are for c, and (x0, y0, z0) are the coordinates of the point with the relatively minimum value.

[0075] Specifically, according to the standard rendering pipeline, the camera transforms the input geometry to obtain mesh normals, image space coordinates, and visually relevant depth, and calculates the color for a given {vertex appearance, mesh normal, lighting condition} according to the lighting and material models; both of these modules are differentiable. However, due to discrete sampling operations, rasterization (which can be regarded as a binary mask determined by the relative positions between pixels and triangles) and z-buffering in the standard graphics pipeline are non-differentiable with respect to image space coordinates and visually relevant depth. Z-buffering merges the rasterization results in a pixel-level one-hot manner based on the relative depth of triangles.

[0076] S3: Combine the probability map to simulate the influence of triangles on the image plane, and use an aggregation function to merge the color maps to obtain the rendering output based on the probability map and relative depth. It should also be noted that:

[0077] Use the probability map Dj to simulate the influence of triangle fj on the image plane; to estimate the probability of Dj at pixel Pi, the function needs to consider both the relative position and distance between Pi and Dj; define Dj at pixel Pi as follows:

[0078]

[0079] where σ is a positive scalar that controls the sharpness of the probability distribution, and δ i j is the sign indicator, δ i j = {+1, if Pi ∈ fi; -1, otherwise}, and d(i,j) is the closest distance from Pi to the edge of triangle fj.

[0080] Refer to Figure 2 to represent the probability maps of triangles with different positive scalars using the Euclidean distance, Figure 2 where (a) in Figure 2 represents the definition of the distance from a pixel to a triangle;

[0081] Specifically, the aggregation function includes:

[0082]

[0083] where C b is the background color, the weight w j = w b = 1, C j is the color map, I is the rendering output, and A S ({C j}) is the aggregation function.

[0084]

[0085] where z i j represents the normalized inverse depth of the 3D point on the triangle fi, whose 2D projection is Pi; ε is a small constant to enable the background color, and γ controls the sharpness of the aggregation function.

[0086] It should also be noted in this embodiment that existing OpenDR technology and NMR technology both use a standard graphics renderer in the forward process, so they cannot control the intermediate rendering process and cannot flow the gradient into the occluded triangles in the final rendered image; while the method of the present invention fully controls the internal variables and can flow the gradient into the invisible triangles through the aggregation function.

[0087] Furthermore, due to the continuous probability formula, the gradient of the pixel Pj in screen space can flow to all distant vertices. However, for traditional OpenDR technology, due to local filtering operations, vertices can only receive gradients from adjacent pixels within a short distance. For traditional NMR technology, there is no definition of the gradient relative to the triangle vertices from the pixels within the white area.

[0088] S4: Reconstruct a single-view mesh and image-based shape fitting. It should also be noted in this step that reconstructing a single-view mesh includes:

[0089] Given an input image, the shape and color generator generates a triangular mesh and its corresponding color;

[0090] Input it into the software rasterizer;

[0091] The soft ray layer simultaneously renders the contour and the color image and provides a rendering-based error signal by comparing with the ground truth.

[0092] Referring to Figure 3 , its framework defines three loss functions, namely the contour loss Ls, the color loss Lc, and the geometric loss Lg, as follows:

[0093]

[0094]

[0095] The final loss is the weighted sum of the three losses:

[0096]

[0097] Image-based shape fitting plays an important role in pose estimation, shape alignment, model-based reconstruction, etc. Traditional methods must rely on rough correspondences, such as 2D joints or feature points, to obtain monitoring signals for optimization; while the differentiable rendering channels designed in the present invention can directly backpropagate pixel-level errors to 3D attributes, thereby achieving dense image-to-3D correspondence, and further achieving high-quality shape fitting.

[0098] However, there are two problems with differentiable renderers, occlusion and long-distance effects. In order to facilitate its application in this embodiment, the probability maps of all triangles are fused through a defined aggregation function, enabling the gradient to flow to all vertices, including occluded vertices. The soft approximation based on probability distribution allows the gradient to propagate to the far end while well controlling the size of the receptive field, thus solving these two problems.

[0099] Generally speaking, the image-based shape fitting problem is accurately solved by minimizing the objective function as follows:

[0100]

[0101] Among them, R(M(p,θ,t)) is the rendering function for the mesh M to generate the rendered image, which is parameterized by the pose θ, translation t, and non-rigid deformation parameter p, and I is the rendering output.

[0102] It should also be elaborated in detail in this embodiment that understanding and reconstructing three-dimensional scenes and structures from two-dimensional images is one of the basic goals of computer vision. The key to three-dimensional reasoning based on images lies in finding sufficient supervision from pixels to three-dimensional attributes; in order to obtain the correlation from images to three dimensions, prior methods mainly rely on matching losses based on two-dimensional key points / contours or shapes / appearances. However, these methods are either only applicable to specific fields or can only provide weak supervision. But in this embodiment, by inverting the renderer, dense pixel-level supervision for general 3D reasoning tasks can be obtained, which cannot be achieved by traditional methods.

[0103] Preferably, in traditional graphics channels, the rendering process is not differentiable. Especially in standard mesh renderers, there is a discrete sampling operation called rasterization, which prevents the gradient from flowing into the mesh vertices. Using a standard graphics renderer directly in the forward process will lead to uncontrolled optimization behavior and limited generalization ability for other three-dimensional reasoning tasks. Therefore, a differentiable rendering framework is designed in the present invention, which can render colored meshes in the forward process.

[0104] Preferably, in this embodiment, an encoder-decoder architecture is used for single-view mesh reconstruction. The encoder is used as a feature extractor with an output feature size of 512. The two networks share the same feature extractor. The shape generator (contained in the feature extractor) consists of three fully connected layers and outputs a displacement vector for each vertex to deform a template mesh into a target model. The color generator contains two parts: a sampling network for sampling the input image to construct a color palette, and a selection network for selecting colors from the color palette to texture the sampled points.

[0105] It should be recognized that embodiments of the present invention can be implemented or carried out by computer hardware, a combination of hardware and software, or by computer instructions stored in a non-transitory computer-readable memory. The methods can be implemented in a computer program using standard programming techniques - including a non-transitory computer-readable storage medium configured with the computer program, wherein the storage medium so configured causes the computer to operate in a specific and predefined manner - according to the methods and drawings described in the specific embodiments. Each program can be implemented in a high-level procedural or object-oriented programming language to communicate with the computer system. However, if desired, the program can be implemented in assembly or machine language. In any case, the language can be a compiled or interpreted language. Additionally, for this purpose, the program is capable of running on a programmed application-specific integrated circuit.

[0106] Furthermore, the operations of the processes described herein can be performed in any suitable order, unless otherwise indicated herein or otherwise clearly contradicted by the context. The processes described herein (or variations and / or combinations thereof) can be executed under the control of one or more computer systems configured with executable instructions and can be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) executed commonly on one or more processors, by hardware, or a combination thereof. The computer program includes a plurality of instructions executable by one or more processors.

[0107] Further, the method can be implemented in any type of computing platform operatively connected to a suitable one, including but not limited to personal computers, minicomputers, mainframes, workstations, network or distributed computing environments, separate or integrated computer platforms, or communicating with charged particle tools or other imaging devices, etc. Aspects of the present invention can be implemented in machine-readable code stored on a non-transitory storage medium or device, whether removable or integrated into the computing platform, such as a hard disk, optical read and / or write storage medium, RAM, ROM, etc., such that it can be read by a programmable computer and, when the storage medium or device is read by the computer, can be used to configure and operate the computer to perform the processes described herein. Additionally, the machine-readable code, or portions thereof, can be transmitted via wired or wireless networks. When such media includes instructions or programs that implement the above-described steps in conjunction with a microprocessor or other data processor, the invention described herein includes these and other different types of non-transitory computer-readable storage media. When programmed according to the methods and techniques of the present invention, the present invention also includes the computer itself. The computer program is capable of applying to input data to perform the functions described herein, thereby transforming the input data to generate output data stored in non-volatile memory. The output information can also be applied to one or more output devices such as a display. In a preferred embodiment of the present invention, the transformed data represents physical and tangible objects, including a specific visual depiction of the physical and tangible objects generated on the display.

[0108] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some or all of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A medical image processing method based on 3D visualization, characterized in that: It includes, Based on the matching loss of key points of medical two-dimensional images for prior, obtaining the correlation between the medical two-dimensional image and the three-dimensional image; Using the correlation to design a differentiable rendering channel, reconstructing a single-view mesh and image-based shape fitting; The reconstruction of the single-view mesh includes, Given an input image, a shape and color generator generates a triangular mesh and its corresponding color; Input it into a soft rasterizer; The soft ray layer simultaneously renders the contour and the color image, and provides a rendering-based error signal by comparing with the ground truth; Combining the probability map to simulate the influence of triangles on the image plane, and using an aggregation function to merge the color map to obtain a rendering output based on the probability map and relative depth.

2. The medical image processing method based on 3D visualization according to claim 1, characterized in that: The image-based shape fitting includes, Among them, R(M(p,θ,t)) is the rendering function for the mesh M to generate a rendered image, which is parameterized by the pose θ, translation t, and non-rigid deformation parameter p, and I is the rendering output.

3. The medical image processing method based on 3D visualization according to claim 1, characterized in that: Designing a differentiable rendering channel using the correlation includes defining external variables for the environmental settings and internal attributes describing the specific properties of the model; The external variables include the camera and lighting conditions; The internal attributes include a triangular mesh and vertex appearance.

4. The medical image processing method based on 3D visualization according to claim 3, characterized in that: The vertex appearance includes color and material.

5. The medical image processing method based on 3D visualization according to claim 1, characterized in that: The combination of the probability map to simulate the influence of triangles on the image plane includes, where σ is a positive scalar that controls the sharpness of the probability distribution, and δ i j is a sign indicator, and δ i j = {+1, if P i ∈ f j ; -1, otherwise}, and d(i, j) is the shortest distance from P i to the triangular f j edge.

6. The medical image processing method based on 3D visualization according to claim 5, characterized in that: The aggregation function includes, Among them, C b is the background color, and the weight w j = w b = 1, C j is the color mapping, I is the rendering output, and A S ({C j}) is the aggregation function; where z i j represents the normalized inverse depth of the 3D point on triangle f j whose 2D projection is P i ; ε is a small constant to enable the background color, and γ controls the sharpness of the aggregation function.

Images