Method for generating internal structure of organism on basis of deep learning

Abstract

The present invention relates to the technical fields of life sciences and medical image devices, and particularly relates to a method for generating the internal structure of an organism on the basis of deep learning. The method comprises the following steps: S1, constructing and training an internal organ generation model of an imaging target; S2, acquiring a three-dimensional body surface contour image of the imaging target; S3, inputting the acquired three-dimensional body surface contour image of the imaging target into the internal organ generation model, so as to obtain a three-dimensional internal organ distribution image; and S4, fusing and displaying the three-dimensional internal organ distribution image and the three-dimensional body surface contour image. By means of the deep-neural-network-based internal organ generation model of the imaging target, it is only necessary to acquire the three-dimensional body surface contour image of the imaging target, and the three-dimensional internal organ distribution image of the imaging target can be quickly and accurately acquired after the three-dimensional body surface contour image is input into the model, thereby simplifying operation steps, realizing automatic imaging and improving the imaging efficiency. Therefore, the method has a strong generalization performance.

Description

A method for generating the internal structure of organisms based on deep learning

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to Chinese patent application filed on December 8, 2023, with application number 202311674971.X, entitled “A method for generating the internal structure of an organism based on deep learning”, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present invention relates to the technical field of life science and medical imaging equipment, and in particular to a method for generating the internal structure of an organism based on deep learning. Background Art

[0004] Biomedical surface imaging is a technology that captures and analyzes images of the surface of living organisms. Current surface imaging technologies mainly include laser scanning imaging, time-of-flight imaging, stereoscopic imaging, and structured light imaging.

[0005] Laser imaging uses a laser beam to scan an object, obtaining three-dimensional surface information. Its primary application in medical imaging is scanning biological objects with laser beams to obtain precise image information. The basic principle of this technology is to scan an object's surface with a laser beam and measure the timing and intensity of the laser beam reflected from the surface to determine the surface's morphology and position.

[0006] Time-of-flight imaging systems consist of an image sensor, an image processing chip, and a modulated light source. Simply put, these systems work by illuminating a scene with a modulated light source and then measuring the phase difference of the reflected wave. Because the speed of light is constant, a time-of-flight imaging system can calculate the distance to each point in the scene based on the time it takes for light to return to the camera. Instead of scanning the image line by line, a time-of-flight imaging system sequentially illuminates the entire scene and then measures the phase difference of the light reflected back to the image sensor. Time-of-flight imaging systems can be used for 3D image acquisition with a large field of view, long distance, low precision, and low cost. Their advantages include fast detection speed, a large field of view, a long working distance, and low price, but they suffer from low precision and susceptibility to interference from ambient light.

[0007] Stereoscopic vision literally means perceiving three-dimensional structures with one or both eyes. Generally speaking, it involves reconstructing the 3D structure or depth information of an object by acquiring two or more images from different viewpoints. Visual cues for depth perception can be categorized as ocular cues and binocular cues. Currently, stereoscopic 3D vision can be achieved through monocular vision, binocular vision, multi-ocular vision, and light field 3D imaging (electronic compound eyes or array cameras).

[0008] Structured light imaging is an imaging technology based on the principle of 3D reconstruction. It projects a specific structured light pattern (such as stripes or grids) onto the surface of the object being imaged. A camera then uses the data to record the deformation of the structured light on the surface, thereby inferring the 3D shape of the object. Structured light imaging typically consists of three main components: a projection system, a camera, and a computer processing system. The projection system typically illuminates the object being measured by projecting a grating or stripe pattern. The shape and size of these gratings or stripes can be adjusted as needed. The camera records the structured light pattern on the surface of the object being measured and converts it into a digital image. The camera's resolution and acquisition speed significantly influence imaging accuracy and real-time performance. The computer processing system processes the acquired image data and reconstructs the 3D shape of the object being measured based on the structured light deformation information. This process typically includes steps such as image preprocessing, camera calibration, 3D reconstruction, and data visualization. Structured light imaging technology offers the advantages of being non-contact, highly precise, and efficient, and is widely used in industrial manufacturing, medical imaging, cultural heritage conservation, and virtual reality.

[0009] However, current biomedical surface imaging systems are unable to present the internal organ structure and distribution of an organism when performing surface imaging. Obtaining the internal organ structure and distribution requires the use of CT images, magnetic resonance images, or other images that can present the internal structure of the organism, which reduces the efficiency of imaging.

[0010] Therefore, there is an urgent need to provide a method for generating the internal structure of an organism based on deep learning, which, compared with the existing technology, can realize automatic imaging of the internal organ structure and distribution of an organism during surface imaging.

[0011] Summary of the Invention

[0012] The present invention solves the technical problems existing in the prior art and provides a method for generating the internal structure of an organism based on deep learning.

[0013] To achieve the above object, the technical solution adopted by the present invention is as follows:

[0014] A method for generating the internal structure of an organism based on deep learning, comprising the following steps:

[0015] S1. Build and train a generative model of the internal organs of the imaging target.

[0016] S2, obtaining a three-dimensional body surface contour image of the target to be imaged;

[0017] S3, inputting the acquired three-dimensional body surface contour image of the imaging target into the internal organ generation model to obtain a three-dimensional image of the internal organ distribution;

[0018] S4, superimposing and fusing the three-dimensional image of internal organ distribution and the three-dimensional body surface contour image, and displaying the image;

[0019] S1 specifically includes the following steps:

[0020] S101, acquiring an original three-dimensional body surface contour image and an internal structure image of an imaging target;

[0021] S102, outlining the internal structure image to obtain a binary mask image of the organ;

[0022] S103, acquiring an organ mask image registered with the original three-dimensional body surface contour image;

[0023] S104, normalizing the original 3D body surface contour image, setting the processed 3D body surface contour image as the input image of the deep learning neural network, and the organ mask image as the output result of the deep learning neural network, and using the 3D body surface contour image and the organ mask image as a training data sample pair;

[0024] S105: Using the training data samples, a neural network-based generation model of the internal organs of the imaging target is trained.

[0025] Furthermore, the training method of the internal organ generation model in step S105 is: the deep learning neural network adopts a diffusion model architecture, and the diffusion model first performs a forward process and then a reverse diffusion process when working;

[0026] The forward process is a process of gradually adding Gaussian noise, specifically adding random noise to the organ mask image of the imaging target;

[0027] The inverse diffusion process is a process of learning the random noise components on the imaging target organ mask under the guidance of the three-dimensional body surface contour image and denoising the imaging target organ mask.

[0028] Furthermore, the forward process is expressed as follows:

[0029] In the above formula, N represents the three-dimensional body surface contour image, Represents a hyperparameter, with a value range of (0,1) and satisfies: t represents a Gaussian noise at a certain time, x t represents a data sample with Gaussian noise at time t, and I represents the identity matrix.

[0030] Furthermore, the reverse diffusion process is a Gaussian distribution process, which is specifically expressed by the following formula:

[0031] p(x t-1 |x t, Q)=N(x t-1 ;Q;μ θ x t-1 ;∑ θ (x t , t))

[0032] In the above formula, N represents the three-dimensional body surface contour image, μ θ ,∑ θ (x t , t) represent learning parameters, and Q represents the three-dimensional body surface contour image obtained by the normalization process in S104.

[0033] Furthermore, the specific method for obtaining the binary mask image of the organ in step S102 is: outlining the organ contour of the imaging target in the internal structure image, and assigning a value of 1 to the area outside the organ contour and a value of 0 to the area inside the contour, thereby obtaining a binary mask image of the target organ.

[0034] Furthermore, the specific method for obtaining the organ mask image in step S103 is: aligning the binary mask image with the original three-dimensional body surface contour image to obtain a contour registration displacement field, and applying the contour registration displacement field to the binary mask image to obtain the organ mask image aligned with the three-dimensional body surface contour image.

[0035] Furthermore, the method for aligning the binary mask image with the original three-dimensional body surface contour image is as follows: first, for the original three-dimensional body surface contour image and the organ mask image of the same cross-section, the cross-section of the original three-dimensional body surface contour image is followed by a body surface contour curve, and the edge of the organ mask image is elastically aligned with the body surface contour curve to obtain an elastic alignment displacement field, and then the elastic alignment displacement field is applied to the organ structure inside the organ mask image.

[0036] Furthermore, the method for aligning the binary mask image with the original three-dimensional body surface contour image is: performing surface-to-surface alignment between the original three-dimensional body surface contour image and the three-dimensional contour of the binary mask image to obtain an elastic alignment displacement field, and then applying the elastic alignment displacement field to the organ structure inside the organ mask image.

[0037] Furthermore, the original three-dimensional body surface contour image is normalized by the following formula:

[0038] In the above formula, Q j represents the jth normalized 3D body surface contour image, P j represents the jth original 3D body surface contour image, represents the minimum value in the jth original three-dimensional body surface contour image, Represents the maximum value in the j-th original 3D body surface contour image.

[0039] Compared with the prior art, the present invention has the following beneficial effects:

[0040] The present invention generates a model of the internal organs of the imaging target based on a deep neural network. It only needs to obtain a three-dimensional surface contour image of the imaging target. After inputting the model, a three-dimensional image of the internal organ distribution of the imaging target can be quickly and accurately obtained. No additional internal structure imaging operations are required, which simplifies the operation steps, realizes automatic imaging, improves imaging efficiency, and has high generalizability. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] FIG1 is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0042] The technical solution of the present invention will be clearly described below in conjunction with the accompanying drawings. Obviously, the described embodiments are not all embodiments of the present invention. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0043] As shown in FIG1 , the present invention provides a method for generating the internal structure of an organism based on deep learning, comprising the following steps:

[0044] S1. Build and train a generative model for the internal organs of the imaging target, specifically including the following steps:

[0045] S101 : Acquire an original three-dimensional body surface contour image and an internal structure image of an imaging target.

[0046] Specifically, the original three-dimensional surface contour image of the imaging target can be obtained through optical surface imaging or other methods such as radar; the internal structure image can be obtained through CT images, magnetic resonance images or other images that can present the internal structure of the imaged organism.

[0047] S102. Outline and process the internal structure image to obtain a binary mask image of the organ: Outline the organ contour of the imaging target in the internal structure image obtained in step S101, and assign a value of 1 to the area outside the organ contour and a value of 0 to the area inside the contour, thereby obtaining a binary mask image of the target organ.

[0048] S103. Obtain an organ mask image aligned with the original three-dimensional body surface contour image: align the binary mask image with the original three-dimensional body surface contour image obtained in step S101 to obtain a contour registration displacement field, apply the obtained contour registration displacement field to the binary mask image, and finally obtain an organ mask image aligned with the original three-dimensional body surface contour image.

[0049] Specifically, the registration method involves first performing elastic registration on the original 3D body contour image and organ mask image of the same cross-section. Following the cross-section of the original 3D body contour image is a body contour curve, the edges of the organ mask image are elastically registered with this curve to obtain an elastic registration displacement field. This elastic registration displacement field is then applied to the organ structure within the organ mask image, ultimately yielding a registered organ mask image. Alternatively, the original 3D body contour image and the 3D contour of the binary mask image can be directly surface-aligned to obtain an elastic registration displacement field, which is then applied to the organ structure within the organ mask image.

[0050] S104, data processing of model training samples: normalizing the original three-dimensional body surface contour image to obtain a normalized three-dimensional body surface contour image. The normalization formula is as follows:

[0051] In the above formula, Q j represents the jth normalized 3D body surface contour image, P j represents the jth original 3D body surface contour image, represents the minimum value in the jth original three-dimensional body surface contour image, Represents the maximum value in the j-th original 3D body surface contour image.

[0052] The normalized three-dimensional body surface contour image is denoted as N. The normalized three-dimensional body surface contour image N is used as the input image of the deep learning neural network, and the organ mask image of the imaging target after registration is used as the output result. The three-dimensional body surface contour image and the corresponding organ mask image of the imaging target are used as a training data sample pair.

[0053] S105. Use the training samples to train a model for generating internal organs of the imaging target: Construct a model for generating internal organs of the imaging target based on a deep neural network, and input the training samples into the deep neural network for training, wherein the deep neural network adopts a diffusion model architecture. When the diffusion model is working, random noise is added to the organ mask image of the imaging target, and the random noise component on the organ mask of the imaging target is learned under the guidance of the three-dimensional body surface contour image. Then, the organ mask of the imaging target is denoised to obtain the relevant imaging target organ structure.

[0054] Specifically, the diffusion model is derived from equilibrium thermodynamics. A Markov chain with a diffusion step is set. By gradually adding random noise to the real data, this process is called the forward process, and then learning the reverse denoising process, which is called the reverse diffusion process, the required data sample results are obtained from the noise. The forward process is a process of gradually adding Gaussian noise. For example, the input data distribution is x~q(x). In the forward process, a total of T times of Gaussian noise are added, thereby generating a series of data samples with Gaussian noise x1, x2, x3, ..., x t ,…,x T ; The forward process is expressed by the following expression:

[0055] In the above formula, Represents a hyperparameter, with a value range of (0,1) and satisfies: t represents Gaussian noise of a certain order, and I represents the identity matrix.

[0056] The inverse diffusion process is the process of gradually recovering the original image (organ mask image) from Gaussian noise under the guidance of the three-dimensional body surface contour image. In the forward process, a small amount of Gaussian noise is added each time, so the inverse diffusion process can also be regarded as a Gaussian distribution. The entire process can be simulated using a neural network. The inverse diffusion process is expressed by the following formula:

[0057] p(x t-1 |x t , Q)=N(x t-1 ;Q;μ θ x t-1 ;∑ θ (x t ,t)) In the above formula, p(x t-1 |x t , Q) represents the inverse diffusion process, N represents the three-dimensional body surface contour image, μ θ ,∑ θ (x t , t) represent learning parameters, and Q represents the three-dimensional body surface contour image obtained by the normalization process in S104.

[0058] S2. Acquire a three-dimensional body surface contour image of the target to be imaged.

[0059] S3. Generate a three-dimensional image of the distribution of the internal organs of the imaging target using the model: Input the three-dimensional body surface contour image obtained in step S2 into the generation model of the internal organs of the imaging target to obtain an organ mask image of the imaging target, that is, a three-dimensional image of the internal organ distribution.

[0060] S4. Overlaying, fusing, and displaying the three-dimensional image of internal organ distribution and the three-dimensional body surface contour image.

[0061] The present invention generates a model of the internal organs of the imaging target based on a deep neural network. It only needs to obtain a three-dimensional surface contour image of the imaging target. After inputting the model, a three-dimensional image of the internal organ distribution of the imaging target can be quickly and accurately obtained. No additional internal structure imaging operations are required, which simplifies the operation steps, improves the imaging efficiency, and has high generalizability.

[0062] Finally, it should be noted that the above content is only used to illustrate the technical solution of the present invention, rather than to limit the scope of protection of the present invention. Simple modifications or equivalent substitutions of the technical solution of the present invention by ordinary technicians in this field do not deviate from the essence and scope of the technical solution of the present invention.

Claims

1. A method for generating internal structure of an organism based on deep learning, characterized in that: The following steps are involved: S1. Build and train an internal organ generation model of the imaging target; S2, obtaining a three-dimensional body surface contour image of the target to be imaged; S3, inputting the acquired three-dimensional body surface contour image of the imaging target into the internal organ generation model to obtain a three-dimensional image of the internal organ distribution; S4, superimposing, fusing and displaying the three-dimensional image of internal organ distribution and the three-dimensional body surface contour image; S1 specifically includes the following steps: S101, acquiring an original three-dimensional body surface contour image and an internal structure image of an imaging target; S102, outlining the internal structure image to obtain a binary mask image of the organ; S103, obtaining an organ mask image registered with the original three-dimensional body surface contour image, specifically, registering the binary mask image with the original three-dimensional body surface contour image to obtain a contour registration displacement field, applying the contour registration displacement field to the binary mask image, and obtaining the organ mask image registered with the three-dimensional body surface contour image; The method for registering the binary mask image with the original three-dimensional body surface contour image is as follows: first, for the original three-dimensional body surface contour image and the organ mask image of the same section, after the section of the original three-dimensional body surface contour image is a body surface contour curve, the edge of the organ mask image is elastically registered with the body surface contour curve to obtain an elastic registration displacement field, and then the elastic registration displacement field is applied to the organ structure inside the organ mask image; Alternatively, the method for registering the binary mask image with the original three-dimensional body surface contour image is: performing surface registration between the original three-dimensional body surface contour image and the three-dimensional contour of the binary mask image to obtain an elastic registration displacement field, and then applying the elastic registration displacement field to the organ structure inside the organ mask image; S104, normalizing the original 3D body surface contour image, setting the processed 3D body surface contour image as an input image of the deep learning neural network, the organ mask image as an output result of the deep learning neural network, and taking the 3D body surface contour image and the organ mask image as a training data sample pair; S105, using the training data samples to train a neural network-based imaging target internal organ generation model.

2. The method for generating the internal structure of an organism based on deep learning according to claim 1, characterized in that: The training method of the internal organ generation model in step S105 is: the deep learning neural network adopts a diffusion model architecture, and the diffusion model first performs a forward process and then a reverse diffusion process when working; The forward process is a process of gradually adding Gaussian noise, specifically adding random noise to the organ mask image of the imaging target; The inverse diffusion process is a process of learning the random noise components on the imaging target organ mask and denoising the organ mask of the imaging target under the guidance of the three-dimensional body surface contour image.

3. The method for generating the internal structure of an organism based on deep learning according to claim 2, characterized in that: The forward process is expressed as follows: In the above formula, N represents the three-dimensional body surface contour image, represents a hyperparameter, the value range is (0,1), and satisfies: t represents a Gaussian noise at a certain time, x t represents a data sample with Gaussian noise at time t, and I represents the identity matrix.

4. The method for generating the internal structure of an organism based on deep learning according to claim 2, characterized in that: The reverse diffusion process is a Gaussian distribution process, which is specifically expressed by the following formula: In the above formula, N represents the three-dimensional body surface contour image, μ θ ,∑ θ (x t , t) all represent learning parameters, and Q represents the three-dimensional body surface contour image obtained by the normalization process in S104.

5. The method for generating the internal structure of an organism based on deep learning according to claim 1, characterized in that: The specific method for obtaining the binary mask image of the organ in step S102 is: outlining the organ contour of the imaging target in the internal structure image, and assigning a value of 1 to the area outside the organ contour and a value of 0 to the area inside the contour, thereby obtaining a binary mask image of the target organ.

6. The method for generating the internal structure of an organism based on deep learning according to claim 1, characterized in that: The normalization process of the original three-dimensional body surface contour image is specifically performed by the following formula: In the above formula, Q j represents the jth normalized 3D body surface contour image, P j represents the jth original 3D body surface contour image, represents the minimum value in the jth original 3D body surface contour image, Represents the maximum value in the jth original 3D body surface contour image.

Images