Image generation method, image processing method, system, device and medium
Image prediction model trained through deep learning algorithms generates three-dimensional or four-dimensional heart structure data from two-dimensional medical images, solving the problem of two-dimensional images restricting the understanding of heart structure in the prior art, improving the accuracy of cardiac analysis and surgical guidance efficiency.
Patent Information
- Application Number
- CN202411238970.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-04
- Publication Date
- 2025-01-03
AI Technical Summary
The two-dimensional images provided by existing intracardiac and transesophageal echocardiography techniques limit the spatial understanding of the heart structure, resulting in the interpretation that depends on the skills and experience of the doctor, is inefficient and prone to misjudgment.
Training image prediction models through deep learning algorithms to generate three-dimensional or four-dimensional heart structure data from two-dimensional medical images, providing a more comprehensive and intuitive high-dimensional structural view.
Improves the accuracy of cardiac analysis, simplifies surgical planning and guidance, optimizes patient care, and improves the efficiency and quality of the processing process.
Smart Images
Figure CN119251331A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of medical imaging technology, and in particular to an image generating method, an image processing method, a system, a device and a medium. Background Art
[0002] Intracardiac ultrasound (ICE) is a minimally invasive imaging technique that uses ultrasound waves to visualize the internal structures of the heart. ICE uses a catheter placed within the heart chamber to provide high-resolution images to guide a variety of cardiac procedures. ICE provides valuable intraoperative guidance through real-time visualization while avoiding exposure to X-ray radiation. Transesophageal echocardiography (TEE) is another imaging modality commonly used during cardiac surgery that can also provide valuable insights. TEE captures images of the heart using an ultrasound probe inserted into the esophagus. TEE typically requires general anesthesia or deep sedation, increasing patient discomfort and complexity of the procedure.
[0003] However, ICE and TEE images are usually 2D (two-dimensional), which limits the spatial understanding of complex cardiac structures. In addition, these images only capture a specific part of the heart each time, with a limited field of view, which requires clinicians or operators to virtually reconstruct a large and complete image, a process that is cumbersome and complex; due to the above limitations, the interpretation of ICE and TEE images is heavily dependent on the skills and experience of clinicians or operators, resulting in the entire processing process being highly dependent on the requirements of specific populations, low processing efficiency, and prone to misjudgment and incomplete analysis. Summary of the invention
[0004] The technical problem to be solved by the present disclosure is to overcome the above-mentioned defects in the prior art, and the purpose is to provide an image generation method, an image processing method, a system, a device and a medium.
[0005] The present invention solves the above technical problems through the following technical solutions:
[0006] The present disclosure provides an image generation method, the image generation method comprising:
[0007] Acquire a plurality of target two-dimensional medical images containing preset objects;
[0008] A plurality of the target two-dimensional medical images are input into a pre-trained image prediction model to output three-dimensional and / or four-dimensional target high-dimensional structural data of the preset object.
[0009] Preferably, the preset object includes a heart, and the target high-dimensional structural data includes a high-dimensional heart structure;
[0010] and / or,
[0011] The target two-dimensional medical image includes an ICE image or a TEE image;
[0012] and / or,
[0013] The target high-dimensional structure data includes at least one of three-dimensional or four-dimensional volume images, surface grid structures, and point clouds.
[0014] Preferably, the step of training the image prediction model comprises:
[0015] Obtain several groups of sample training data;
[0016] Wherein, each group of the sample training data includes sample two-dimensional medical images of the preset object at different acquisition perspectives and corresponding sample high-dimensional structure data;
[0017] Obtaining sample preset key content for constructing a high-dimensional structure of the preset object in each sample two-dimensional medical image;
[0018] Based on each sample two-dimensional medical image in each group of the sample training data, as well as the corresponding sample preset key content and the sample high-dimensional structure data, the preset network model is trained to obtain the image prediction model.
[0019] Preferably, the step of obtaining the sample preset key content for constructing the high-dimensional structure of the preset object in each sample two-dimensional medical image comprises:
[0020] Performing segmentation processing on the sample two-dimensional medical image to obtain a plurality of sample segmentation sub-images of different regions of interest on the preset object;
[0021] and / or,
[0022] The sample two-dimensional medical image is analyzed and processed to obtain information of several sample key anatomical points on the preset object.
[0023] Preferably, the step of training the preset network model based on each sample two-dimensional medical image in each group of the sample training data, and the corresponding sample preset key content and the sample high-dimensional structure data to obtain the image prediction model includes:
[0024] Input the sample segmentation sub-image after segmentation of each sample two-dimensional medical image in each group of the sample training data into the preset network model, transfer it to the encoder and decoder structure in the preset network model for training and learning to obtain the corresponding feature map, and obtain the corresponding sample prediction high-dimensional structure data based on the volume rendering of several feature maps;
[0025] Comparing the sample predicted high-dimensional structure data with the corresponding sample high-dimensional structure data to obtain a comparison result, and calculating a loss value based on the comparison result;
[0026] In response to the loss value not satisfying the preset model training condition, the preset network model continues to be trained based on the loss value until the loss value satisfies the preset model training condition.
[0027] Preferably, the step of obtaining the sample two-dimensional medical image comprises:
[0028] While keeping the position of the imaging catheter unchanged, based on moving the tip of the imaging catheter to different positions, the sample two-dimensional medical images of the preset object at different acquisition viewing angles are acquired;
[0029] and / or,
[0030] The sample high-dimensional structural data is obtained based on at least one of three-dimensional or four-dimensional CT (computed tomography), MRI (magnetic resonance imaging), X-ray, and ultrasound;
[0031] and / or,
[0032] The preset network model includes a pre-trained large basic model.
[0033] The present disclosure also provides an image processing method, which is obtained based on the above-mentioned image generation method, and the image processing method comprises:
[0034] Visualize the target high-dimensional structure data of the preset object;
[0035] or,
[0036] The target high-dimensional structure data of the preset object is visualized, and in response to different external operation instructions in the target high-dimensional structure data, the target high-dimensional structure data is controlled to move to the corresponding target display angle for display.
[0037] Preferably, after the step of respectively controlling the target high-dimensional structure data to move to the corresponding target position for display, the method further includes:
[0038] Analyzing the target high-dimensional structure data at different target display angles to obtain analysis results;
[0039] Based on the analysis result, generating treatment-related information matching the current state of the preset object;
[0040] and / or,
[0041] The image processing method further comprises:
[0042] Three-dimensional printing is performed based on the target high-dimensional structure data of the preset object to obtain the three-dimensional printed structure of the preset object.
[0043] The present disclosure also provides an image generation system, the image generation system comprising:
[0044] A target image acquisition module, used to acquire a plurality of target two-dimensional medical images containing a preset object;
[0045] The image generation module is used to input a plurality of the target two-dimensional medical images into a pre-trained image prediction model to output three-dimensional and / or four-dimensional target high-dimensional structure data of the preset object.
[0046] Preferably, the preset object includes a heart, and the target high-dimensional structural data includes a high-dimensional heart structure;
[0047] and / or,
[0048] The target two-dimensional medical image includes an ICE image or a TEE image;
[0049] and / or,
[0050] The target high-dimensional structure data includes at least one of three-dimensional or four-dimensional volume images, surface grid structures, and point clouds.
[0051] Preferably, the image generation system further includes a model training module, and the model training module includes:
[0052] A training data acquisition unit, used to acquire several groups of sample training data;
[0053] Wherein, each group of the sample training data includes sample two-dimensional medical images of the preset object at different acquisition perspectives and corresponding sample high-dimensional structure data;
[0054] A sample content acquisition unit, used to acquire sample preset key content for constructing a high-dimensional structure of the preset object in each sample two-dimensional medical image;
[0055] A model training unit is used to train a preset network model based on each sample two-dimensional medical image in each group of the sample training data, as well as the corresponding sample preset key content and the sample high-dimensional structure data, so as to obtain the image prediction model.
[0056] Preferably, the sample content acquisition unit is further used to perform segmentation processing on the sample two-dimensional medical image to obtain a plurality of sample segmentation sub-images of different regions of interest on the preset object;
[0057] and / or,
[0058] The sample two-dimensional medical image is analyzed and processed to obtain information of several sample key anatomical points on the preset object.
[0059] Preferably, the model training unit includes:
[0060] A sample data acquisition subunit, used for inputting the sample segmentation sub-image after segmentation of each sample two-dimensional medical image in each group of the sample training data into the preset network model, transferring it to the encoder and decoder structure in the preset network model for training and learning to obtain the corresponding feature map, and obtaining the corresponding sample prediction high-dimensional structure data based on the volume rendering of several of the feature maps;
[0061] A loss value calculation subunit, used for comparing the sample prediction high-dimensional structure data with the corresponding sample high-dimensional structure data to obtain a comparison result, and calculating a loss value based on the comparison result;
[0062] The model training subunit is used to continue training the preset network model based on the loss value in response to the loss value not satisfying the preset model training condition until the loss value satisfies the preset model training condition.
[0063] Preferably, the sample data acquisition subunit is further used for acquiring the sample two-dimensional medical images of the preset object at different acquisition viewing angles by moving the tip of the imaging catheter to different positions while keeping the imaging catheter position unchanged;
[0064] and / or,
[0065] The sample high-dimensional structural data is obtained based on at least one of three-dimensional or four-dimensional CT, MRI, X-ray, and ultrasound;
[0066] and / or,
[0067] The preset network model includes a pre-trained large basic model.
[0068] The present disclosure further provides an image processing system, which is obtained based on the above-mentioned image generation system, and comprises:
[0069] A display module is used to visualize the target high-dimensional structure data of the preset object;
[0070] or,
[0071] The display module is used to visualize the target high-dimensional structure data of the preset object and call the instruction response module;
[0072] The instruction response module is used to respond to different external operation instructions in the target high-dimensional structure data, and call the display module to control the target high-dimensional structure data to move to the corresponding target display angle for display.
[0073] Preferably, the image processing system further includes:
[0074] A data analysis module, used to analyze the target high-dimensional structure data at different target display angles to obtain analysis results;
[0075] A correlation information generating module, used for generating treatment correlation information matching the current state of the preset object based on the analysis result;
[0076] and / or,
[0077] The image processing system further comprises:
[0078] A printing control module is used to perform three-dimensional printing based on the target high-dimensional structure data of the preset object to obtain a three-dimensional printed structure of the preset object.
[0079] The present disclosure also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and used to run on the processor, characterized in that the processor implements the above-mentioned image generation method or the above-mentioned image processing method when executing the computer program.
[0080] The present disclosure also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the above-mentioned image generating method or the above-mentioned image processing method when executed by a processor.
[0081] The present disclosure also provides a computer program product, including a computer program, characterized in that when the computer program is executed by a processor, it implements the above-mentioned image generation method, or implements the above-mentioned image processing method.
[0082] On the basis of being in accordance with the common sense in the art, the above-mentioned preferred conditions can be arbitrarily combined to obtain the preferred embodiments of the present disclosure.
[0083] The positive and progressive effects of this disclosure are:
[0084] In the present disclosure, a solution for generating high-dimensional cardiac structure from two-dimensional images such as ICE or TEE is proposed. The image prediction model is trained based on a deep learning algorithm, which can provide a more comprehensive and intuitive high-dimensional structural view of the internal anatomical structure of the heart, thereby effectively improving the analysis accuracy of preset objects such as the heart, improving surgical planning and guidance, and optimizing overall patient care, etc., simplifying the workload and operation complexity of operators, greatly improving the efficiency and quality of the entire processing flow, and ensuring the user experience of operators and patients. BRIEF DESCRIPTION OF THE DRAWINGS
[0085] Figure 1 This is a flowchart of the image generation method of embodiment 1 of the present disclosure.
[0086] Figure 2 This is a flowchart of the image generation method of embodiment 2 of the present disclosure.
[0087] Figure 3 This is a schematic diagram of the model training structure of Example 2 of the present disclosure.
[0088] Figure 4 Schematic diagram of images collected from different viewing angles in Example 2 of the present disclosure.
[0089] Figure 5 This is a flowchart of the image processing method of embodiment 3 of the present disclosure.
[0090] Figure 6 A flowchart of an image processing method according to a fourth embodiment of the present disclosure.
[0091] Figure 7 A schematic diagram of the modules of the image generation system of Embodiment 5 of the present disclosure.
[0092] Figure 8 A schematic diagram of the modules of the image generation system of Embodiment 6 of the present disclosure.
[0093] Fig. 9 This is a flowchart of the image processing system of Embodiment 7 of the present disclosure.
[0094] Fig.10 This is a flowchart of the image processing system of Example 8 of the present disclosure.
[0095] Fig.11 This is a schematic diagram of the structure of an electronic device according to Embodiment 9 of the present disclosure. DETAILED DESCRIPTION
[0096] The present disclosure is further described below by way of examples, but the present disclosure is not limited to the scope of the examples.
[0097] Prefixes such as "first" and "second" are used in the embodiments of the present disclosure only to distinguish different description objects, and have no limiting effect on the position, order, priority, quantity or content of the described objects. The use of prefixes such as ordinal numbers to distinguish description objects in the embodiments of the present disclosure does not constitute a limitation on the described objects. For the statement of the described objects, please refer to the description in the context of the claims or embodiments, and no unnecessary limitation should be constituted due to the use of such prefixes. In addition, in the description of the present embodiment, unless otherwise specified, the meaning of "plurality" is two or more.
[0098] In the embodiments of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.
[0099] Example 1
[0100] like Figure 1 As shown, the image generation method of this embodiment includes:
[0101] S101, obtaining a plurality of target two-dimensional medical images containing preset objects;
[0102] Among them, the preset objects include parts such as the heart;
[0103] The target two-dimensional medical images include but are not limited to ICE images and TEE images.
[0104] Specifically, the plurality of target two-dimensional medical images may be a two-dimensional medical image of a preset object of a patient at a certain acquisition angle, or may be multiple two-dimensional medical images of a preset object from the same patient at different acquisition angles.
[0105] S102: Input a plurality of target two-dimensional medical images into a pre-trained image prediction model to output three-dimensional and / or four-dimensional target high-dimensional structure data of a preset object.
[0106] Among them, the preset network model includes but is not limited to a pre-trained large basic model.
[0107] In this embodiment, a scheme for generating high-dimensional cardiac structure from two-dimensional images such as ICE or TEE is proposed. The image prediction model is trained based on a deep learning algorithm, and can provide a more comprehensive and intuitive high-dimensional structural view of the internal anatomical structure of the heart, thereby effectively improving the analysis accuracy of preset objects such as the heart, improving surgical planning and guidance, and optimizing overall patient care, etc., simplifying the workload and operation complexity of operators, greatly improving the efficiency and quality of the entire processing flow, and ensuring the user experience of operators and patients.
[0108] Example 2
[0109] The image generation method of this embodiment is a further improvement on Embodiment 1, specifically:
[0110] In one implementable solution, the target high-dimensional structural data includes three-dimensional or four-dimensional volume images, surface mesh structures, point clouds, etc.
[0111] In the present disclosure, after a two-dimensional medical image is input into an image prediction model, one or more 3D or 4D structural contents matching a preset object (such as a heart) in the two-dimensional medical image can be output to meet the high-dimensional image analysis requirements of different actual scenarios, so that operators can more conveniently and comprehensively observe the problems existing in the current preset object, simplifying the workload and operation complexity of doctors and related personnel while ensuring the accuracy and efficiency of subsequent image analysis.
[0112] In one feasible solution, Figure 2 As shown, the image prediction model needs to be trained before step S102. The specific steps of training the image prediction model include:
[0113] S1001, obtaining several groups of sample training data;
[0114] Each set of sample training data includes sample two-dimensional medical images of preset objects at different acquisition angles and corresponding sample high-dimensional structure data;
[0115] The sample high-dimensional structural data includes three-dimensional or four-dimensional volume images, surface grid structures, point clouds, etc. corresponding to the preset objects.
[0116] In one feasible solution, the high-dimensional structural data of the sample is obtained based on three-dimensional or four-dimensional CT, MRI, X-ray, ultrasound, etc.
[0117] Specifically, the sample training data in the training data set can be obtained from a phantom (i.e., a physical phantom or a digital phantom), that is, a two-dimensional medical image and paired three-dimensional structural data are obtained from the phantom; or, synthetic data can be generated from a 3D heart model, that is, two-dimensional medical images corresponding to 3D objects at different perspectives are obtained to obtain sample training data at different perspectives.
[0118] S1002, obtaining sample preset key content for constructing a high-dimensional structure of a preset object in each sample two-dimensional medical image;
[0119] S1003. Based on each sample two-dimensional medical image in each group of sample training data, as well as the corresponding sample preset key content and sample high-dimensional structure data, the preset network model is trained to obtain an image prediction model.
[0120] In this scheme, an image prediction model is obtained by training with a number of sample training data, so that for any two-dimensional medical image containing a preset object, the high-dimensional structure corresponding to the preset object can be directly and accurately predicted; the current image prediction model can be updated with new sample training data based on the model update rules (such as regular or irregular) to further ensure the accuracy and reliability of the high-dimensional structure.
[0121] In one feasible solution, step S1002 includes:
[0122] Segment the sample two-dimensional medical image to obtain several sample segmentation sub-images of different regions of interest on the preset object;
[0123] When the preset object is the heart, segmentation includes identifying and outlining a single heart structure, such as segmenting different regions of interest: left ventricle, right ventricle, atrium and valves, etc. If there are other images such as CT, MRI, etc., the heart structure of the region of interest can also be segmented. Among them, image segmentation can be performed manually, semi-automatically or fully automatically, and the specific method can be determined or adjusted according to the actual scene requirements.
[0124] In this scheme, other redundant information such as body parts is removed through image segmentation, that is, the key content of the subsequent model training process is extracted from the sample two-dimensional medical image to avoid introducing unnecessary information into the model training process, thereby ensuring the efficiency of model training and the reliability of the model.
[0125] In one feasible solution, step S1002 includes:
[0126] The sample two-dimensional medical image is analyzed and processed to obtain information of several sample key anatomical points on the preset object.
[0127] In this scheme, key anatomical points, such as the sinus ostium and oval fossa, are detected on the sample two-dimensional medical image; the anatomical points can be used to establish correspondence when matching different perspectives or across image modalities. These anatomical points can be manually selected by the operator, or obtained semi-automatically or fully automatically through an image analysis algorithm. The specific method can be determined or adjusted according to the actual scene requirements.
[0128] In this scheme, the key anatomical points in each image are detected through image analysis algorithms, and other redundant information such as body parts are removed. That is, the key content of the subsequent model training process is extracted from the sample two-dimensional medical images to avoid introducing unnecessary information into the model training process, thereby ensuring the efficiency of model training and the reliability of the model.
[0129] In one feasible solution, step S1003 includes:
[0130] Input the sample segmentation sub-image after segmentation of each sample two-dimensional medical image in each group of sample training data into the preset network model, pass it to the encoder and decoder structure in the preset network model for training and learning to obtain the corresponding feature map, and obtain the corresponding sample prediction high-dimensional structure data based on the volume rendering of several feature maps;
[0131] Compare the sample prediction high-dimensional structure data with the corresponding sample high-dimensional structure data to obtain a comparison result, and calculate the loss value based on the comparison result;
[0132] In response to the loss value not satisfying the preset model training condition, the preset network model is continuously trained based on the loss value until the loss value satisfies the preset model training condition.
[0133] Specifically, the training dataset is passed to a deep learning framework to train the intrinsic relationship between the two-dimensional medical image representation and the underlying high-dimensional cardiac structure; the goal of the deep learning model is to learn the mapping function:
[0134]
[0135] Among them, (x, y, z,) is the view position, (α, β) is the view angle, and ρ is the intensity value. The intensity value ρ can be binary, multi-label, or normalized raw ultrasound image value. For simplicity, if a high-dimensional object is constructed based on a segmented image (e.g., right atrium), a binary value (0 for background and 1 for object) can be used; the input two-dimensional medical image is passed to an encoder and decoder structure in the model to learn feature maps, which will be used for volume rendering to generate predicted images; the predicted image is compared with the input image to calculate the loss value, which can be determined based on pixel-level intensity differences or surface distances, such as Figure 3 shown.
[0136] Among them, 3DObject represents the three-dimensional structure of the heart, input Image represents the sample two-dimensional medical image of the input model, features represents features, Projected Image represents the projected image, Loss represents the loss value, and Reconstruction represents the image reconstruction operation, so as to train a model for outputting the high-dimensional structure corresponding to the two-dimensional medical image.
[0137] Among them, a pre-trained large-scale basic model can be used to generate features; of course, other available methods can also be used, which are not specifically limited here.
[0138] In one feasible solution, step S1001 includes:
[0139] While keeping the position of the imaging catheter unchanged, sample two-dimensional medical images of the preset object at different acquisition viewing angles are acquired based on different orientations of the tip of the moving imaging catheter.
[0140] When the imaging catheter is inserted and advanced into or near the heart, when the tip of the catheter approaches the heart during this process, corresponding images can be acquired to obtain multi-view ICE or TEE images, etc., for constructing corresponding high-dimensional objects.
[0141] Ideally, the catheter tip should be in a relatively fixed position to obtain multi-view images at the same reference point to ensure the accuracy and reliability of subsequent model predictions. The position can be determined manually by the clinician or semi-automatically or fully automatically through image analysis algorithms. For example, when the catheter enters the heart to collect images through ICE or reaches a specific position with a good cardiac view to collect images through TEE, the clinician can be prompted to lock the catheter while only allowing the catheter tip to rotate to provide a relatively fixed position to obtain multi-view images; Figure 4 As shown, the figure shows two ICE images (two corresponding to the upper and lower positions) acquired at different viewing angles at the same catheter tip position.
[0142] In addition, other images such as CT, MRI, X-ray, ultrasound, etc. can also be combined with the above-mentioned acquired images to improve the reconstruction process; these additional images can be obtained before or during surgery.
[0143] In this solution, the acquisition equipment is ensured to be in the same position through automatic, semi-automatic or fully automatic methods, so that multi-view images can be obtained at the same reference point, effectively ensuring the accuracy and reliability of subsequent model predictions; in addition, other types of two-dimensional medical images can be combined with sample images for model training to further improve the accuracy and reliability of model predictions.
[0144] Example 3
[0145] like Figure 5 As shown, the image processing method of this embodiment is obtained based on the above-mentioned image generation method, and the image processing method includes:
[0146] S201, visually displaying target high-dimensional structure data of a preset object;
[0147] or,
[0148] S202 , visually display the target high-dimensional structure data of the preset object, and in response to different external operation instructions in the target high-dimensional structure data, control the target high-dimensional structure data to move to the corresponding target display angle for display.
[0149] In this solution, any target two-dimensional medical image is input into the image prediction model to directly generate the corresponding high-dimensional cardiac structure representation, which can be visualized on a dedicated workstation. The operator can also flexibly manipulate the 3D object at any time to view it from different angles according to actual needs, so as to understand the cardiac anatomical structure more comprehensively and intuitively, optimize the operator's operating method, simplify the operator's workload and operation complexity, and effectively improve the operator and patient experience throughout the operation process.
[0150] Example 4
[0151] The image processing method of this embodiment is a further improvement on Embodiment 3. Specifically:
[0152] In one feasible solution, Figure 6 As shown, after step S202, the following steps are also included:
[0153] S203, analyzing the target high-dimensional structure data at different target display angles to obtain analysis results;
[0154] S204: Generate treatment-related information that matches the current state of the preset object based on the analysis result.
[0155] In this solution, the three-dimensional or four-dimensional structures corresponding to different display angles correspond to different information. The high-dimensional structural state at each perspective can be obtained through manual or specific analysis algorithms to analyze and obtain relevant recommended information on the current heart's treatment, surgical planning, nursing, etc., providing more accurate information support, effectively ensuring a more accurate analysis and judgment of the current patient's heart, making the entire processing process more intelligent and convenient.
[0156] In an implementable solution, the image processing method further includes:
[0157] Three-dimensional printing is performed based on the target high-dimensional structure data of the preset object to obtain the three-dimensional printed structure of the preset object.
[0158] like Figure 7 As shown, the image generation system of the present disclosure includes:
[0159] A target image acquisition module 1 is used to acquire a plurality of target two-dimensional medical images containing a preset object;
[0160] Among them, the preset objects include parts such as the heart;
[0161] The target two-dimensional medical images include but are not limited to ICE images and TEE images.
[0162] The image generation module 2 is used to input a plurality of target two-dimensional medical images into a pre-trained image prediction model to output three-dimensional and / or four-dimensional target high-dimensional structure data of a preset object.
[0163] Among them, the preset network model includes but is not limited to a pre-trained large basic model.
[0164] As for the system embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The system embodiment described above is only illustrative, wherein the units described as separate components may or may not be physically separated, and the components as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the disclosed solution.
[0165] Example 6
[0166] like Figure 8 As shown, the image generation system of this embodiment is a further improvement of Embodiment 5, specifically:
[0167] In one implementable solution, the target high-dimensional structural data includes three-dimensional or four-dimensional volume images, surface mesh structures, point clouds, etc.
[0168] In an implementable solution, the image generation system further includes a model training module 3, and the model training module 3 includes:
[0169] A training data acquisition unit 4 is used to acquire several groups of sample training data;
[0170] Each set of sample training data includes sample two-dimensional medical images of preset objects at different acquisition angles and corresponding sample high-dimensional structure data;
[0171] A sample content acquisition unit 5 is used to acquire sample preset key content for constructing a high-dimensional structure of a preset object in each sample two-dimensional medical image;
[0172] The model training unit 6 is used to train the preset network model based on each sample two-dimensional medical image in each group of sample training data, as well as the corresponding sample preset key content and sample high-dimensional structure data, so as to obtain an image prediction model.
[0173] In an implementable solution, the sample content acquisition unit 5 is further used to perform segmentation processing on the sample two-dimensional medical image to obtain a plurality of sample segmentation sub-images of different regions of interest on the preset object;
[0174] and / or,
[0175] The sample two-dimensional medical image is analyzed and processed to obtain information of several sample key anatomical points on the preset object.
[0176] In an implementable solution, the model training unit 6 includes:
[0177] The sample data acquisition subunit is used to input the sample segmentation sub-image after segmentation of each sample two-dimensional medical image in each group of sample training data into the preset network model, pass it to the encoder and decoder structure in the preset network model for training and learning to obtain the corresponding feature map, and obtain the corresponding sample prediction high-dimensional structure data based on the volume rendering of several feature maps;
[0178] A loss value calculation subunit is used to compare the sample prediction high-dimensional structure data with the corresponding sample high-dimensional structure data to obtain a comparison result, and calculate a loss value based on the comparison result;
[0179] The model training subunit is used to continue training the preset network model based on the loss value in response to the loss value not satisfying the preset model training condition until the loss value satisfies the preset model training condition.
[0180] In an implementable solution, the sample data acquisition subunit is further used to acquire sample two-dimensional medical images of a preset object at different acquisition viewing angles based on different positions of the tip of the moving imaging catheter while keeping the imaging catheter in position.
[0181] As for the system embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The system embodiment described above is only illustrative, wherein the units described as separate components may or may not be physically separated, and the components as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the disclosed solution.
[0182] Example 7
[0183] The image processing system disclosed in the present invention is obtained based on the above-mentioned image generation system.
[0184] like Fig. 9 As shown, the image processing system comprises:
[0185] Display module 7, used for visually displaying target high-dimensional structure data of a preset object;
[0186] or,
[0187] Display module 7, used for visually displaying the target high-dimensional structure data of the preset object and calling the instruction response module;
[0188] The instruction response module 8 is used to respond to different external operation instructions in the target high-dimensional structure data, and call the display module 7 to control the target high-dimensional structure data to move to the corresponding target display angle for display.
[0189] As for the system embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The system embodiment described above is only illustrative, wherein the units described as separate components may or may not be physically separated, and the components as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the disclosed solution.
[0190] Example 8
[0191] The image processing system of this embodiment is a further improvement on Embodiment 3. Specifically:
[0192] like Fig.10 As shown, the image processing system also includes:
[0193] The data analysis module 9 is used to analyze the target high-dimensional structure data under different target display angles to obtain analysis results;
[0194] The associated information generating module 10 is used to generate treatment associated information matching the current state of the preset object based on the analysis result.
[0195] The image processing system also includes:
[0196] The printing control module is used to perform three-dimensional printing based on the target high-dimensional structure data of the preset object to obtain the three-dimensional printed structure of the preset object.
[0197] As for the system embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The system embodiment described above is only illustrative, wherein the units described as separate components may or may not be physically separated, and the components as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the disclosed solution.
[0198] Example 9
[0199] Fig.11This is a schematic diagram of the structure of an electronic device showing an example embodiment of the present disclosure, the electronic device includes a memory, a processor, and a computer program stored in the memory and used to run on the processor, and the processor implements the method described in any of the above embodiments when executing the computer program. Fig.10 The electronic device 90 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.
[0200] like Fig.10 As shown, the electronic device 90 may be in the form of a general-purpose computing device, for example, it may be a server device. The components of the electronic device 90 may include, but are not limited to: at least one processor 91, at least one memory 92, and a bus 93 connecting different system components (including the memory 92 and the processor 91).
[0201] The bus 93 includes a data bus, an address bus, and a control bus.
[0202] The memory 92 may include a volatile memory, such as a random access memory (RAM) 921 and / or a cache memory 922 , and may further include a read-only memory (ROM) 923 .
[0203] The memory 92 may also include a program tool 925 (or utility) having a set (at least one) of program modules 924, such program modules 924 including but not limited to: an operating system, one or more application programs, other program modules and program data, each of which or some combination may include an implementation of a network environment.
[0204] The processor 91 executes various functional applications and data processing by running the computer program stored in the memory 92, such as the method provided in any of the above embodiments.
[0205] The electronic device 90 may also communicate with one or more external devices 94 (e.g., keyboards, pointing devices, etc.). Such communication may be performed via an input / output (I / O) interface 95. Furthermore, the electronic device 90 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 96. As shown, the network adapter 96 communicates with other modules of the electronic device 90 via a bus 93. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 90, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID (disk array) systems, tape drives, and data backup storage systems, etc.
[0206] It should be noted that although several units / modules or sub-units / modules of the electronic device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more units / modules described above can be embodied in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided into multiple units / modules to be embodied.
[0207] Example 10
[0208] The embodiments of the present disclosure further provide a computer-readable storage medium having a computer program stored thereon, and when the program is executed by a processor, the method provided in any of the above embodiments is implemented.
[0209] The readable storage medium may include but is not limited to: a portable disk, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory, an optical storage device, a magnetic storage device or any suitable combination of the above.
[0210] Embodiment 11
[0211] The present disclosure also provides a computer program product, including a computer program, which implements any of the above methods when executed by a processor.
[0212] Among them, the program code for executing the computer program product of the present disclosure can be written in any combination of one or more programming languages, and the program code can be executed completely on the user device, partially on the user device, as an independent software package, partially on the user device and partially on a remote device, or completely on the remote device.
[0213] Although the specific embodiments of the present disclosure are described above, those skilled in the art should understand that this is only an example, and the protection scope of the present disclosure is defined by the appended claims. Those skilled in the art may make various changes or modifications to these embodiments without departing from the principles and essence of the present disclosure, but these changes and modifications all fall within the protection scope of the present disclosure.
Claims
1. An image generation method, characterized in that: The image generation method comprises: Acquire a plurality of target two-dimensional medical images containing preset objects; A plurality of the target two-dimensional medical images are input into a pre-trained image prediction model to output three-dimensional and / or four-dimensional target high-dimensional structural data of the preset object.
2. The image generation method according to claim 1, characterized in that: The preset object includes a heart, and the target high-dimensional structure data includes a high-dimensional heart structure; and / or, The target two-dimensional medical image includes an ICE image or a TEE image; and / or, The target high-dimensional structure data includes at least one of three-dimensional or four-dimensional volume images, surface grid structures, and point clouds.
3. The image generation method according to claim 1 or 2, characterized in that: The step of training the image prediction model comprises: Obtain several groups of sample training data; Wherein, each group of the sample training data includes sample two-dimensional medical images of the preset object at different acquisition perspectives and corresponding sample high-dimensional structure data; Obtaining sample preset key content for constructing a high-dimensional structure of the preset object in each sample two-dimensional medical image; Based on each sample two-dimensional medical image in each group of the sample training data, as well as the corresponding sample preset key content and the sample high-dimensional structure data, the preset network model is trained to obtain the image prediction model.
4. The image generation method according to claim 3, characterized in that: The step of obtaining the sample preset key content for constructing the high-dimensional structure of the preset object in each sample two-dimensional medical image comprises: Performing segmentation processing on the sample two-dimensional medical image to obtain a plurality of sample segmentation sub-images of different regions of interest on the preset object; and / or, The sample two-dimensional medical image is analyzed and processed to obtain information of several sample key anatomical points on the preset object.
5. The image generation method according to claim 3, characterized in that: The step of training the preset network model based on each sample two-dimensional medical image in each group of the sample training data, and the corresponding sample preset key content and the sample high-dimensional structure data to obtain the image prediction model includes: Input the sample segmentation sub-image after segmentation of each sample two-dimensional medical image in each group of the sample training data into the preset network model, transfer it to the encoder and decoder structure in the preset network model for training and learning to obtain the corresponding feature map, and obtain the corresponding sample prediction high-dimensional structure data based on the volume rendering of several feature maps; Comparing the sample predicted high-dimensional structure data with the corresponding sample high-dimensional structure data to obtain a comparison result, and calculating a loss value based on the comparison result; In response to the loss value not satisfying the preset model training condition, the preset network model continues to be trained based on the loss value until the loss value satisfies the preset model training condition.
6. The image generation method according to claim 3, characterized in that: The step of obtaining the sample two-dimensional medical image comprises: While keeping the position of the imaging catheter unchanged, based on moving the tip of the imaging catheter to different positions, the sample two-dimensional medical images of the preset object at different acquisition viewing angles are acquired; and / or, The sample high-dimensional structural data is obtained based on at least one of three-dimensional or four-dimensional CT, MRI, X-ray, and ultrasound; and / or, The preset network model includes a pre-trained large basic model.
7. An image processing method, characterized in that: The image processing method is obtained based on the image generation method according to any one of claims 1 to 6, and the image processing method comprises: Visualize the target high-dimensional structure data of the preset object; or, The target high-dimensional structure data of the preset object is visualized, and in response to different external operation instructions in the target high-dimensional structure data, the target high-dimensional structure data is controlled to move to the corresponding target display angle for display.
8. The image processing method according to claim 7, characterized in that: After the step of respectively controlling the target high-dimensional structure data to move to the corresponding target position for display, the method further includes: Analyzing the target high-dimensional structure data at different target display angles to obtain analysis results; Based on the analysis result, generating treatment-related information matching the current state of the preset object; and / or, The image processing method further comprises: Three-dimensional printing is performed based on the target high-dimensional structure data of the preset object to obtain the three-dimensional printed structure of the preset object.
9. An image generation system, characterized in that: The image generation system comprises: A target image acquisition module, used to acquire a plurality of target two-dimensional medical images containing a preset object; The image generation module is used to input a plurality of the target two-dimensional medical images into a pre-trained image prediction model to output three-dimensional and / or four-dimensional target high-dimensional structure data of the preset object.
10. An image processing system, characterized in that: The image processing system is obtained based on the image generation system according to claim 9, and the image processing system comprises: A display module is used to visualize the target high-dimensional structure data of the preset object; or, The display module is used to visualize the target high-dimensional structure data of the preset object and call the instruction response module; The instruction response module is used to respond to different external operation instructions in the target high-dimensional structure data, and call the display module to control the target high-dimensional structure data to move to the corresponding target display angle for display.
11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and used to run on the processor, characterized in that: When the processor executes the computer program, the image generating method described in any one of claims 1 to 6 is implemented, or the image processing method described in claim 7 or 8 is implemented.
12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the image generating method according to any one of claims 1 to 6 is implemented, or the image processing method according to claim 7 or 8 is implemented.
13. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the image generating method according to any one of claims 1 to 6 is implemented, or the image processing method according to claim 7 or 8 is implemented.