Ultrasonic image prediction and correction method and system based on multi-modal image data and medium

Abstract

The invention discloses an ultrasonic image prediction and correction method and system based on multi-modal image data and a medium, and the method comprises the steps: constructing a three-dimensional model of a target body based on the multi-modal image data, the multi-modal image data comprises at least two of a CT image, a CTA image and an ultrasonic image; determining a space coordinate system corresponding to the three-dimensional model as a first coordinate system; acquiring a CT image of the target body, determining a space coordinate system of the CT image as a second coordinate system, and determining a mapping relation from the first coordinate system to the second coordinate system; simulating an ultrasonic probe to probe the three-dimensional model, and determining a probing section; determining the position coordinates of the probing section in the first coordinate system, and mapping the position coordinates of the probing section to the second coordinate system according to the mapping relation to obtain a CT section; and predicting an ultrasonic image corresponding to the probing section according to the CT section. According to the method, the corresponding ultrasonic image can be predicted according to the CT image, so that the quality of the ultrasonic image in the operation process is improved.

Description

Technical Field

[0001] The present invention relates to the field of medical device technology, and in particular to an ultrasonic image prediction and correction method, system and medium based on multimodal imaging data. Background Art

[0002] In interventional surgery, especially complex surgery on important organs such as the liver and heart, accurate navigation and positioning are crucial to the success of the surgery. Traditional interventional surgery usually relies on a single imaging mode, but the limitations of a single imaging mode can easily lead to insufficient information, increasing the difficulty and risk of the surgery.

[0003] Multimodal information fusion technology can combine the advantages of different imaging methods, make up for the shortcomings of a single imaging technology, and achieve a more comprehensive and accurate assessment of the surgical area. At the same time, ultrasound, as a real-time imaging technology, has unique advantages in interventional surgery and can provide real-time feedback on tissue changes in the patient's body. However, the existing ultrasound guidance system is not sufficiently integrated with other imaging technologies and cannot provide comprehensive decision support.

[0004] Traditional interventional procedures usually rely on a single imaging modality (such as ultrasound, CTA or CT), which has the following disadvantages:

[0005] (1) Information limitations: A single imaging mode can only provide limited tissue information. For example:

[0006] CT: It can provide high-resolution three-dimensional imaging, but it is usually a static image acquired before surgery and lacks real-time dynamic feedback during surgery.

[0007] CTA: It is mainly used to display the morphological information of vascular structures, such as vascular stenosis, aneurysm, thrombus, etc. Although the pathological conditions of blood vessels can be displayed in detail, the information about other soft tissues, nerves, muscles or surrounding organs is relatively insufficient, so it is difficult to provide a comprehensive anatomical view.

[0008] Ultrasound: Although it has the advantage of real-time imaging, its image quality is easily affected by gas, bone or fat tissue in the patient's body, the resolution and contrast are relatively low, and the imaging effect of deep tissue is poor.

[0009] (2) Lack of real-time dynamic information

[0010] Static imaging modes (such as CT or CTA) can only provide the patient's anatomical structure before surgery. However, the body structure may change during surgery (such as tissue displacement and organ deformation). A single imaging mode cannot update this information in real time, causing doctors to rely on preoperative images to guide the surgery, which may not be consistent with the actual situation during the operation.

[0011] (3) Unable to integrate multiple information

[0012] A single imaging method can usually only capture a specific type of information (such as real-time dynamic ultrasound or high-resolution anatomical images of CT), and cannot fully reflect multiple aspects of complex anatomical areas. For interventional surgeries that require consideration of multiple structures such as blood vessels, nerves, and tissues, a single imaging mode is prone to missing key information, increasing surgical risks.

[0013] (4) Large error

[0014] Due to the limitations of a single imaging mode, doctors must rely on personal experience to make up for the shortcomings of imaging technology when operating. This subjective judgment may lead to errors, especially when dealing with complex interventional procedures, the accuracy and precision of positioning may not be enough, increasing the risk of misoperation or complications.

[0015] (5) Difficulty in identifying key anatomical structures

[0016] Some key anatomical structures (such as small blood vessels, nerve fibers, etc.) may not be clearly presented in a single imaging mode, especially the unclear boundary identification of soft tissues and tumors, which increases the risk of accidental injury to important structures during surgery.

[0017] (6) Limitations of imaging technology

[0018] Each imaging technique has its inherent physical limitations. For example:

[0019] CT requires a long time to scan and process, and it is difficult to provide real-time dynamic feedback during surgery.

[0020] CTA relies on the use of contrast agents to enhance the visibility of blood vessels; contrast agent injection rate, dose, and the patient's circulatory status may also affect the quality and accuracy of CTA images.

[0021] Ultrasound is easily affected by artifacts and images may contain noise.

[0022] The disclosure of the above background technology content is only used to assist in understanding the inventive concept and technical solution of the present invention. It does not necessarily belong to the prior art of this patent application, nor does it necessarily provide technical guidance. In the absence of clear evidence that the above content has been disclosed before the filing date of this patent application, the above background technology should not be used to evaluate the novelty and creativity of the present application. Summary of the invention

[0023] The purpose of the present invention is to provide an ultrasound image prediction and correction method, system and medium based on multimodal imaging data, which can predict ultrasound images according to CT images, so as to improve the quality of real ultrasound images and provide support for decision-making during interventional surgery.

[0024] To achieve the above object, the technical solution adopted by the present invention is as follows:

[0025] A method for predicting an ultrasound image based on multimodal imaging data, the method comprising the following steps:

[0026] Building a three-dimensional model of the target body based on multimodal image data, wherein the multimodal image data includes at least two of CT images, CTA images, and ultrasound images, and determining that a spatial coordinate system corresponding to the three-dimensional model is a first coordinate system;

[0027] Acquire a CT image of the target body, determine that a spatial coordinate system corresponding to the CT image is a second coordinate system, and determine a mapping relationship from the first coordinate system to the second coordinate system;

[0028] Based on the three-dimensional model, simulating an ultrasonic probe to explore the three-dimensional model, and determining a section of the three-dimensional model to be explored by the ultrasonic probe under target ultrasonic imaging requirements;

[0029] Determine the position coordinates of the exploration section in the first coordinate system, and map the position coordinates of the exploration section to the second coordinate system according to a mapping relationship from the first coordinate system to the second coordinate system to obtain a CT section;

[0030] An ultrasonic image corresponding to the exploratory section is predicted according to the CT section, and the ultrasonic image is an ultrasonic prediction image.

[0031] Further, based on any one of the technical solutions or a combination of multiple technical solutions described above, the target ultrasound imaging requirement is an optimal ultrasound imaging requirement, and the exploration section determined based on the optimal ultrasound imaging requirement is the optimal exploration section;

[0032] The three-dimensional model includes virtual skin tissue, soft tissue and bones, and the optimal exploration section is determined by:

[0033] Simulating an ultrasonic probe to explore the three-dimensional model in different positions and postures to obtain a plurality of simulated ultrasonic images;

[0034] Performing an evaluation operation on each simulated ultrasound imaging to obtain an evaluation result, wherein the evaluation operation includes air-skin tissue interface evaluation and / or soft tissue-bone interface evaluation;

[0035] Determine the optimal evaluation result and its corresponding optimal simulated ultrasonic imaging from the evaluation results corresponding to the respective simulated ultrasonic imaging;

[0036] The posture of the ultrasound probe corresponding to the optimal simulated ultrasound imaging is taken as the optimal posture, and the optimal exploration section corresponding to the ultrasound probe in the optimal posture is determined.

[0037] Further, based on any one of the above technical solutions or a combination of multiple technical solutions, the air-skin tissue interface assessment comprises the following steps:

[0038] Determining a projection direction of the ultrasound probe relative to the three-dimensional model according to the position and posture of the ultrasound probe, and calculating a projection area of ​​the ultrasound probe on the surface of the skin tissue along the projection direction;

[0039] Determining a surface area of ​​the skin tissue corresponding to the projection area as a projection surface, and performing meshing processing on the projection surface to generate a polygonal mesh;

[0040] For each vertex on the polygonal mesh, determine a line segment from the vertex along the opposite direction of the projection direction to the surface of the ultrasound probe, and calculate the length of the line segment;

[0041] The air-skin tissue interface was evaluated based on the length of all line segments.

[0042] Further, based on any one of the above-mentioned technical solutions or a combination of multiple technical solutions, determining the air-skin tissue interface evaluation result according to the length of all line segments includes the following steps:

[0043] Calculate the average and variance of the lengths of all line segments, and count the percentage of line ends whose lengths are less than the preset length to the total number of line ends;

[0044] The air-skin tissue interface is evaluated according to the mean value, variance and percentage to obtain an air-skin tissue interface evaluation result, wherein the air-skin tissue interface evaluation result is negatively correlated with the mean value and the variance, and the air-skin tissue interface evaluation result is positively correlated with the percentages.

[0045] Further, based on any one of the above technical solutions or a combination of multiple technical solutions, the soft tissue-bone interface assessment comprises the following steps:

[0046] Determine the location of the ultrasonic sound source of the ultrasonic probe according to the position of the ultrasonic probe, and determine a sector-shaped area within the detection section of the ultrasonic probe with the ultrasonic sound source as the center of the circle;

[0047] The estimated value of the bone occlusion ultrasound in the fan-shaped area is calculated, and the soft tissue-bone interface is evaluated according to the estimated value of the bone occlusion ultrasound to obtain a soft tissue-bone interface evaluation result. The smaller the estimated value of the bone occlusion ultrasound, the better the soft tissue-bone interface evaluation result.

[0048] Further, based on any one of the above-mentioned technical solutions or a combination of multiple technical solutions, the estimated value of the bone-blocked ultrasound is calculated in the following manner:

[0049] Arrange a plurality of rays extending toward the three-dimensional model with the center of the circle as an endpoint in the fan-shaped area, and the plurality of rays are evenly distributed in the fan-shaped area;

[0050] According to the principle of ultrasonic imaging, a weight value is configured for each of the rays based on Gaussian normal distribution;

[0051] Determine the intersection line segment of each ray and the bone, and calculate the estimated value of the ray being blocked by the bone according to the weight value and the intersection line segment;

[0052] The estimated value of the bone-occluded ultrasound is calculated based on the estimated values ​​of all rays being blocked by the bones.

[0053] Further, based on any one of the above-mentioned technical solutions or a combination of multiple technical solutions, the calculation formula of the estimated value of the bone occlusion ultrasound is expressed as follows:

[0054]

[0055] Wherein, WAvg is the estimated value of the bone blocking ultrasound, m is the number of the rays, j represents a positive integer from 1 to m, Lj represents the total length of the intersection line segment between the jth ray and the bone, θ represents the angle between the jth ray and the center line of the ultrasound probe, and σ represents the standard deviation.

[0056] Further, based on any one of the above-mentioned technical solutions or a combination of multiple technical solutions, predicting the ultrasound image corresponding to the exploration section according to the CT section includes the following steps:

[0057] Inputting the data corresponding to the CT section into a pre-constructed ultrasound image prediction model to output a predicted ultrasound image corresponding to the exploratory section;

[0058] The ultrasound image prediction model is established by the following steps:

[0059] Collecting a learning sample set, each sample in the learning sample set includes a CT image and an ultrasound image of the same object;

[0060] Designing a basic model, the basic model comprising an encoder, a decoder and a reconstruction loss calculation module, wherein the encoder is configured to extract image features of the CT image, the decoder is configured to generate a pseudo ultrasound image according to the image features, and the reconstruction loss calculation module is configured to calculate the reconstruction loss of the pseudo ultrasound image;

[0061] The basic model is trained using a preset loss function so that the pseudo ultrasound image generated by the trained model based on the CT image is consistent with the real ultrasound image, thereby obtaining the ultrasound image prediction model.

[0062] Further, based on any one of the above-mentioned technical solutions or a combination of multiple technical solutions, the reconstruction loss calculation module is configured to calculate the reconstruction loss of the pseudo ultrasound image according to the following formula:

[0063] L Rec =||R.US-P.US|| 2

[0064] Among them, L Rec represents the reconstruction loss, R.US represents the real ultrasound image, and P.US represents the pseudo ultrasound image.

[0065] Furthermore, any one of the above-mentioned technical solutions or a combination of multiple technical solutions further includes a discriminator, which is configured to judge the authenticity of each area in the pseudo ultrasound image, and its calculation formula is:

[0066] L Dis =log(Dis(R.US))+log(1-(Dis(P.US)))

[0067] Among them, L Dis Represents the authenticity calculation result of the region, Dis(R.US) represents the output result of the region discriminator with the ultrasonic image as input, and Dis(P.US) represents the output result of the region discriminator with the pseudo ultrasonic image as input.

[0068] Further, based on any one of the above technical solutions or a combination of multiple technical solutions, the basic model also includes a quantization module. During the training process of the basic model, the quantization module is configured to quantize the image features to obtain quantized encoding information;

[0069] The decoder is configured to generate the pseudo ultrasound image according to the quantized encoding information.

[0070] Further, based on any one of the above-mentioned technical solutions or a combination of multiple technical solutions, the calculation formula of the quantization loss is expressed as follows:

[0071]

[0072] Among them, L q represents the quantization loss, E c represents the image features, detach() represents the separation and replication of the gradient of the feature vector during model training, E qrepresents the quantized coding information.

[0073] Further, based on any one of the above-mentioned technical solutions or a combination of multiple technical solutions, the three-dimensional model is constructed by the following steps:

[0074] Acquiring multimodal image data of the target body, wherein the multimodal image data includes at least two of CT images, CTA images, and ultrasound images;

[0075] Segmenting a target region in the multimodal image data based on a deep learning algorithm, wherein the target region includes one or more of a target organ, a lesion area, a blood vessel, a nerve, and a bone;

[0076] The multimodal image data are registered and integrated through image registration and fusion technology to generate the three-dimensional model.

[0077] According to another aspect of the present invention, the present invention provides an ultrasonic image correction method based on multimodal image data, comprising the following steps:

[0078] For a target object, under the target ultrasound imaging requirement, obtaining a real ultrasound image of the target object;

[0079] Based on the ultrasonic image prediction method based on multimodal image data described in any one of the technical solutions or a combination of multiple technical solutions described above, predict the corresponding predicted ultrasonic image under the target ultrasonic imaging requirement;

[0080] The real ultrasound image is corrected using the predicted ultrasound image.

[0081] According to another aspect of the present invention, the present invention provides an ultrasound image prediction system based on multimodal imaging data, which utilizes the ultrasound image prediction method based on multimodal imaging data as described in any one of the above technical solutions or a combination of multiple technical solutions to predict the corresponding ultrasound image under the target ultrasound imaging requirements based on the CT image of the target body.

[0082] According to another aspect of the present invention, the present invention provides a computer-readable storage medium for storing program instructions, wherein the program instructions are configured to call and execute the steps of the method described in any one of the above technical solutions or a combination of multiple technical solutions.

[0083] The beneficial effects brought by the technical solution provided by the present invention are as follows:

[0084] a. The ultrasound image prediction method based on multimodal imaging data provided by the present invention can simulate ultrasound imaging at different angles and positions by constructing a three-dimensional model of the target body, and predict the ultrasound image at the corresponding position of the ultrasound probe based on the pre-acquired CT image. Based on the predicted ultrasound image, the ultrasound image / data in the actual surgery can be corrected and compared, reducing the impact of image artifacts and noise, and improving the imaging quality in the surgery;

[0085] b. The present invention simulates ultrasound imaging at different angles and positions, comprehensively evaluates the effects of the air-skin tissue interface and the soft tissue-bone interface on ultrasound imaging, can determine the exploration section corresponding to the best ultrasound imaging, and predict the ultrasound image corresponding to the ultrasound section, so that doctors can better choose the intervention path and the position and posture of the ultrasound probe;

[0086] c. The present invention can automatically and accurately generate corresponding ultrasound prediction images based on CT sections through a pre-constructed ultrasound image prediction model, and enhances the model's ability to capture image structure information by introducing quantization operations. A discriminator is introduced on the basis of the quantized autoencoder to determine the authenticity of the generated samples, thereby optimizing the model's sense of reality and quality at the level of generated details, and ensuring the clarity and detail richness of the predicted ultrasound image. BRIEF DESCRIPTION OF THE DRAWINGS

[0087] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0088] Figure 1 A flowchart of an ultrasound image prediction method based on multimodal imaging data provided by an exemplary embodiment of the present invention;

[0089] Figure 2 An interface diagram of multiple viewing angles of a three-dimensional model provided for an exemplary embodiment of the present invention;

[0090] Figure 3 A simulated interface diagram of an ultrasound probe projected onto the skin of a three-dimensional model provided by an exemplary embodiment of the present invention;

[0091] Figure 4 A simulated interface diagram of a projection surface provided for an exemplary embodiment of the present invention;

[0092] Figure 5A simulation interface diagram of line segment statistics from each vertex on the skin projection to the probe provided by an exemplary embodiment of the present invention;

[0093] Figure 6 A simulated interface diagram of air-skin tissue interface evaluation results provided for an exemplary embodiment of the present invention;

[0094] Figure 7 A flow chart of an air-skin tissue interface assessment process provided for an exemplary embodiment of the present invention;

[0095] Figure 8 A simulation interface diagram of a simulated ultrasonic exploration provided by an exemplary embodiment of the present invention;

[0096] Fig. 9 A graph of a Gaussian normal distribution provided for an exemplary embodiment of the present invention;

[0097] Fig.10 A first-perspective simulation interface diagram of a simulated ultrasound exploration blocked by bones provided by an exemplary embodiment of the present invention;

[0098] Fig.11 A second perspective simulation interface diagram of a simulated ultrasound exploration blocked by bones provided by an exemplary embodiment of the present invention;

[0099] Fig.12 A flow chart of a soft tissue-bone interface assessment process provided for an exemplary embodiment of the present invention;

[0100] Fig.13 A schematic diagram of an ultrasound image prediction model provided for an exemplary embodiment of the present invention. DETAILED DESCRIPTION

[0101] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0102] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, device, product or equipment that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0103] In one embodiment of the present invention, a method for ultrasonic image prediction based on multimodal image data is provided. Figure 1 , the method comprises the following steps:

[0104] Building a three-dimensional model of a target body based on multimodal image data, wherein the three-dimensional model includes virtual skin tissue, soft tissue, and bones, and the multimodal image data includes at least two of CT images, CTA images, and ultrasound images, and determining that a spatial coordinate system corresponding to the three-dimensional model is a first coordinate system;

[0105] Acquire a CT image of the target body, determine that a spatial coordinate system corresponding to the CT image is a second coordinate system, and determine a mapping relationship from the first coordinate system to the second coordinate system;

[0106] Based on the three-dimensional model, simulating an ultrasonic probe to explore the three-dimensional model, and determining a section of the three-dimensional model to be explored by the ultrasonic probe under target ultrasonic imaging requirements;

[0107] Determine the position coordinates of the exploration section in the first coordinate system, and map the position coordinates of the exploration section to the second coordinate system according to a mapping relationship from the first coordinate system to the second coordinate system to obtain a CT section;

[0108] The ultrasonic image corresponding to the exploration section is predicted according to the CT section.

[0109] Among them, in the process of determining the CT section for predicting the ultrasound image according to the exploration section, there are at least the following two methods.

[0110] One way is to directly determine the position coordinates of the exploration section in the first coordinate system, that is, the surface coordinates, and use them as the first surface coordinates. According to the mapping relationship from the first coordinate system to the second coordinate system, the first surface coordinates are mapped to the second coordinate system to obtain the second surface coordinates, and the section on the CT image corresponding to the second surface coordinates is the CT section.

[0111] Another way is to determine the set of position coordinates of the points on the exploration section in the first coordinate system as the first coordinate set. According to the mapping relationship from the first coordinate system to the second coordinate system, the first coordinate set is mapped to the second coordinate system to obtain a second coordinate set; the set of CT image data corresponding to the second coordinate set is the CT section.

[0112] The target ultrasound imaging requirement may be determined manually or automatically. The target ultrasound imaging requirement determined manually may be specified manually. For example, the position of the ultrasound probe is determined manually, and the detection section of the ultrasound probe on the three-dimensional model in the position is the target prediction object. In some specific applications, the detection section of a target may even be directly specified manually.

[0113] In one embodiment of the present invention, a method is proposed to automatically simulate and determine the ultrasonic probe to explore the target body in different postures and different angles and positions and determine the optimal ultrasonic imaging conditions / requirements, using the optimal ultrasonic imaging requirements as the target ultrasonic imaging, and the exploration section determined based on the optimal ultrasonic imaging requirements as the optimal exploration section.

[0114] In this embodiment, the three-dimensional model is first constructed by the following steps:

[0115] Acquiring multimodal image data of the target body, wherein the multimodal image data includes at least two of CT images, CTA images, and ultrasound images;

[0116] Segmenting target areas such as target organs, lesions, blood vessels, nerves, and bones in the multimodal image data based on a deep learning algorithm, for example, using a convolutional neural network to segment the image data of each modality to ensure accuracy and efficiency of the segmentation;

[0117] By using image registration and fusion technology, imaging data from different modalities (such as CT and CTA) are integrated to generate a unified and complete three-dimensional model of the surgical area;

[0118] Based on the three-dimensional model, a high-resolution three-dimensional image is generated to provide doctors with a three-dimensional and intuitive display of the surgical target area, facilitating further image prediction, path planning, and surgical implementation.

[0119] Based on AI and the three-dimensional model of the target object (surgical subject), the imaging effect under ultrasound guidance during the operation is predicted, providing doctors with visual anticipation support during the operation.

[0120] In one embodiment of the present invention, the imaging effect under ultrasound guidance during the operation is predicted in the following manner to determine the optimal exploration section, which mainly includes two parts: ultrasound effect evaluation of the intervention path and ultrasound image prediction.

[0121] Evaluation of ultrasound effect of interventional pathway: Through the patient's individualized three-dimensional model, ultrasound imaging at different angles and positions is simulated, the impact of the air-skin tissue interface and soft tissue-bone interface on ultrasound imaging is comprehensively evaluated, and the ultrasound images that may be obtained during the operation are predicted, so that doctors can better choose the interventional pathway and the position and posture of the ultrasound probe.

[0122] Ultrasound image prediction: The ultrasound image at the corresponding position of the ultrasound probe is predicted based on the pre-acquired CT image. Based on the predicted ultrasound image, the ultrasound image / data in the actual surgery can be corrected and compared to reduce the impact of image artifacts and noise and improve the imaging quality during surgery.

[0123] By combining the patient's CT images and 3D models, and considering the spatial position relationship of the ultrasound probe, based on the principle of ultrasound imaging, this application comprehensively evaluates the impact of the air-skin tissue interface and the soft tissue-bone interface on ultrasound imaging. The ultrasound cross section is generated using CT images, and further AI is used to generate ultrasound prediction images. The process is described in detail below.

[0124] In one embodiment of the present invention, the optimal exploration section is determined by:

[0125] Simulating an ultrasonic probe to explore the three-dimensional model in different positions and postures to obtain a plurality of simulated ultrasonic images;

[0126] Performing an evaluation operation on each simulated ultrasound imaging to obtain an evaluation result, wherein the evaluation operation includes air-skin tissue interface evaluation and / or soft tissue-bone interface evaluation;

[0127] Determine the optimal evaluation result and its corresponding optimal simulated ultrasonic imaging from the evaluation results corresponding to the respective simulated ultrasonic imaging;

[0128] The posture of the ultrasound probe corresponding to the optimal simulated ultrasound imaging is taken as the optimal posture, and the optimal exploration section corresponding to the ultrasound probe in the optimal posture is determined.

[0129] The process and principle of air-skin tissue interface evaluation are shown in Figures 3 to 7 First, the projection direction of the ultrasound probe relative to the three-dimensional model is determined according to the posture of the ultrasound probe. In this embodiment, the projection direction is the orientation direction / ultrasound emission direction of the ultrasound probe, and the projection area of ​​the ultrasound probe on the surface of the skin tissue of the three-dimensional model along the projection direction is calculated.

[0130] The surface area of ​​the skin tissue of the three-dimensional model corresponding to the projection area is determined as the projection surface, and the projection surface is meshed to generate a polygonal mesh. In the meshing process, the projection surface is sampled and a basic polygonal mesh is generated, and the basic polygonal mesh is further triangulated by applying vtkTriangleFilter based on EarCutting to generate a triangle mesh; vtkLinearSubdivisionFilter is applied to linearly subdivide the triangle mesh; and then vtkSmoothPolyDataFilter based on LaplacianSmoothingAlgorithm is applied to make the mesh uniform and smooth to obtain the final polygonal mesh.

[0131] For each vertex on the polygonal mesh, a line segment from the vertex along the opposite direction of the projection direction to the surface of the ultrasound probe is determined, and the length Li of the line segment is calculated.

[0132] After finding the length of each line segment, all line segment samples are statistically analyzed to evaluate the air-skin tissue interface:

[0133] Compute the average length of the line segments:

[0134] Compute the variance of the line segment lengths:

[0135] Calculate the percentage of line segments whose length is less than 1.5 mm:

[0136] Where Li represents the length of the i-th line segment, N represents the total number of all line segments, n represents the number of line segments with a length less than 1.5 mm, avg represents the average value of the line segment length, var represents the variance of the line segment length, and pct represents the percentage of line segments with a length less than 1.5 mm.

[0137] It should be noted that in the present embodiment, the percentage of line segments with a length less than 1.5 mm is counted, while in other embodiments, the percentage of line segments with a length less than 1.2 mm may be counted as needed. Specifically, the corresponding preset length is determined as needed, and the percentage of the number of line ends with a length less than the preset length to the number of bus ends is counted.

[0138] The air-skin tissue interface evaluation result is negatively correlated with the average value, that is, the smaller the average value of the line segment length is, the better the air-skin tissue interface evaluation result is; the air-skin tissue interface evaluation result is negatively correlated with the variance, that is, the smaller the average value of the line segment length is, the better the air-skin tissue interface evaluation result is; the air-skin tissue interface evaluation result and the percentage are positively correlated, that is, the larger the percentage is, the better the air-skin tissue interface evaluation result is.

[0139] See also Figures 8 to 12 , the soft tissue-bone interface evaluation process and principle are as follows.

[0140] The position of the ultrasonic sound source of the ultrasonic probe is determined according to the position of the ultrasonic probe, and a sector-shaped area is determined within the detection section of the ultrasonic probe with the ultrasonic sound source as the center of the circle. Preferably, the center line of the sector-shaped area is the orientation direction / ultrasound emission direction of the ultrasonic probe.

[0141] like Figure 8 As shown, a fan-shaped area with a central angle of 60° is configured with the ultrasonic sound source as the center of the circle and the direction of the ultrasonic probe as the center line. The fan-shaped area intersects with the three-dimensional model, and the radiation depth of the fan-shaped area at least covers the depth of the three-dimensional model that needs to be ultrasonically detected, or in other words, the area on the three-dimensional model that needs to be ultrasonically detected is covered by the fan-shaped area. The ultrasonic direction of the ultrasonic probe is simulated to uniformly configure ("emit") a number of rays in the fan-shaped area of ​​60°, and each ray has the center of the circle as the endpoint and extends toward the three-dimensional model along the ultrasonic emission direction.

[0142] According to the principle of ultrasonic imaging, Fig. 9 The Gaussian normal distribution shown configures weight values ​​for each of the rays. The weight values ​​of the rays are:

[0143]

[0144] Wherein, Wj(θ) represents the weight value of the j-th ray, θ represents the angle between the j-th ray and the center line of the ultrasonic probe, and σ represents the standard deviation.

[0145] See also Fig.10 , Fig.11 , determine the intersection line segment of each ray and the bone, calculate the estimated value of the ray blocked by the bone according to the weight value and the intersection line segment; calculate the estimated value of the bone blocking ultrasound according to the estimated value of all rays blocked by the bone. The calculation formula of the estimated value of the bone blocking ultrasound is expressed as follows:

[0146]

[0147] Wherein, WAvg is the estimated value of the bone occluding ultrasound, m is the number of the rays, j is a positive integer from 1 to m, and Lj represents the total length of the intersection line segment between the jth ray and the bone.

[0148] The soft tissue-bone interface evaluation result is negatively correlated with the estimated value of the bone occlusion ultrasound, that is, the smaller the estimated value of the bone occlusion ultrasound, the better the soft tissue-bone interface evaluation result.

[0149] After comprehensively evaluating the effects of the air-skin tissue interface and the soft tissue-bone interface on ultrasonic imaging and determining the optimal exploration section, a set of position coordinates of points on the optimal exploration section in the first coordinate system is obtained and used as the first coordinate set. According to the mapping relationship from the first coordinate system to the second coordinate system, the position coordinates of points on the optimal exploration section in the first coordinate system are mapped to the second coordinate system to determine the position coordinates of the exploration section to be predicted in the second coordinate system, i.e., the second coordinate set, and the ultrasonic image corresponding to the exploration section is predicted according to the CT screenshot corresponding to the second coordinate set.

[0150] In this embodiment, the CT screenshot corresponding to the second coordinate set is input into a pre-constructed ultrasound image prediction model to output the predicted ultrasound image corresponding to the exploration section.

[0151] In one embodiment of the present invention, the advantages of adversarial generative networks and variational autoencoders are combined to propose a cross-modal medical image style transfer generation model based on adversarial architecture and feature quantization, namely the ultrasound image prediction model, which aims to simultaneously improve the realism and structural consistency of the generated images, thereby providing high-quality ultrasound images during interventional treatment and assisting precise surgical treatment.

[0152] In this embodiment, the ultrasound image prediction model is established through the following steps.

[0153] A learning sample set is collected, each sample in the learning sample set includes a CT image and an ultrasound image of the same object.

[0154] A basic model is designed, and the basic model includes an encoder, a quantization module, a decoder, a reconstruction loss calculation module and a discriminator. The encoder is configured to obtain the encoding information of the CT image, and the quantization module is configured to quantize the encoding information to obtain the quantized encoding information; the decoder is configured to generate a pseudo ultrasonic image according to the quantized encoding information, the reconstruction loss calculation module is configured to calculate the reconstruction loss of the pseudo ultrasonic image, and the discriminator is configured to determine the consistency of the pseudo ultrasonic image with each area of ​​the real ultrasonic image. The present invention is improved on the basis of the probability model, and a quantization module is added, which can realize the quantization of the feature vector in a learnable feature matching manner.

[0155] The learning sample set is input into the basic model, and the basic model is trained using a preset loss function. Fig.13 As shown, for the input CT screenshot (x c ), the basic model first encodes it through the encoder (Encoder) to obtain the corresponding encoding information (E c ∈(C,H,W)). Then in the quantization process, the encoding information E c First, it is expanded into an ordered 2D feature vector form (V c ), and randomly sample a 2D weight vector of equal size from a uniform distribution (V q ). By calculating the shortest Euclidean distance square between the two two-dimensional vectors, the corresponding relationship between them is determined. The calculation formula is as follows:

[0156]

[0157] Among them, index represents the index set of the minimum Euclidean distance square. In the quantization process, the closest feature is matched based on this set, and finally the quantized encoding information (E q ∈(C,H,W)).

[0158] Through the above method, the quantized encoded information can effectively reconstruct the features of the original image, while reducing the risk of mode collapse when the model is trained on a small batch data set. However, since the quantization process contains non-differentiable operations, the present invention deconstructs the quantization process into two parts during back propagation by means of quantization loss, and optimizes it by gradient replication. The calculation formula of the quantization loss is as follows:

[0159]

[0160] Among them, L q represents the quantization loss, E cDetach() represents the encoding information, and detach() represents the separation and replication of the gradient of the feature vector during the model training process so that the encoder and decoder can achieve end-to-end training. q represents the quantized coding information.

[0161] The present invention enhances the model's ability to capture image structural information by introducing quantization operations in the feature space. However, the quantization process may result in deficiencies in the clarity and detail richness of the generated samples. To address this challenge, the present invention introduces a discriminator based on the quantized autoencoder to determine the authenticity of the generated samples, thereby optimizing the model's sense of reality and quality at the level of generated details.

[0162] like Fig.13 As shown in the figure, for the input CT screenshot, a pseudo ultrasound image (Pseudo US) is generated by the autoencoder. First, based on the reconstruction loss (L Rec ) calculates the loss in the generation process, and introduces a new regional discriminator to discriminate the local area of ​​the generated pseudo ultrasound image. Local area judgment means discriminating the local area of ​​the pseudo ultrasound image instead of the global judgment of the entire image, so as to effectively capture the texture and subtle structure in the image. In this way, the generated image appears more realistic in the local area, while reducing the computational complexity, making this method suitable for the task of generating high-resolution images.

[0163] The reconstruction loss calculation module is configured to calculate the reconstruction loss of the pseudo ultrasound image based on a calculation method of the square of the L1 norm, and the calculation formula of the reconstruction loss is:

[0164] L Rec =||R.US-P.US|| 2

[0165] Among them, L Rec represents the reconstruction loss, and R.US represents the real ultrasound image ( Fig.13 P.US represents the pseudo ultrasound image ( Fig.13 Pseudo US in the United States).

[0166] After adding the discriminator, the present invention also introduces the discriminant loss in addition to the quantization loss and reconstruction loss during the training process. For the discriminant loss, the present invention calculates the discriminant loss based on each area of ​​the generated image, and calculates the real ultrasound image and the pseudo ultrasound image through binary cross entropy. The calculation formula is as follows:

[0167] L Dis=log(Dis(R.US))+log(1-(Dis(P.US)))

[0168] Among them, L Dis Represents the authenticity calculation result of the region, Dis(R.US) represents the output result of the region discriminator with the ultrasonic image as input, and Dis(P.US) represents the output result of the region discriminator with the pseudo ultrasonic image as input.

[0169] The basic model is trained using a preset loss function. During the entire training process, the generator and the discriminator perform adversarial training based on the above loss function, thereby continuously optimizing the generation effect of the pseudo ultrasound image, so that the pseudo ultrasound image generated by the trained model based on the CT image is consistent with the real ultrasound image, thereby obtaining the ultrasound image prediction model.

[0170] In one embodiment of the present invention, a method for ultrasonic image correction based on multimodal image data is provided, comprising the following steps:

[0171] For a target object, under the target ultrasound imaging requirement, obtaining a real ultrasound image of the target object;

[0172] Based on the ultrasonic image prediction method based on multimodal image data described in any one of the above embodiments, predict a corresponding predicted ultrasonic image under target ultrasonic imaging requirements;

[0173] The predicted ultrasound image is used to compare and correct the real ultrasound image, thereby reducing artifacts in the ultrasound image during the actual operation and reducing the impact of noise on ultrasound imaging during the actual operation, improving the imaging quality during the operation, and providing doctors with clearer and higher-quality ultrasound images.

[0174] Preferably, the display device / module displays the three-dimensional model under multiple viewing angles, the air-skin tissue interface evaluation results and the soft tissue-bone interface evaluation results, the simulation process of ultrasound imaging under different angles and positions, and the predicted ultrasound image and other information.

[0175] In one embodiment of the present invention, an ultrasound image prediction system based on multimodal imaging data is provided, including a data acquisition module, an AI segmentation and three-dimensional reconstruction module, an ultrasound simulation and evaluation module, an ultrasound imaging prediction module and a display module.

[0176] The data acquisition module is responsible for collecting and integrating the patient's multimodal medical imaging data as the basis for subsequent processing and analysis. The multimodal data collected by the data acquisition module includes imaging technologies such as CT, CTA, and ultrasound. Each imaging technology provides different anatomical and functional information, laying the foundation for comprehensive surgical planning.

[0177] The data acquisition module is equipped with a data preprocessing function, which can perform standardized processing on the collected raw data, such as image noise filtering, image registration, contrast adjustment, etc., to ensure that data from different modalities can be effectively fused.

[0178] The data acquisition module is also equipped with data transmission and storage, which can ensure the efficient transmission and safe storage of large-capacity medical imaging data, and support data retrieval before, during and after surgery.

[0179] The AI ​​segmentation and 3D reconstruction module is configured to automatically segment the multimodal imaging data of the patient using an artificial intelligence algorithm and generate a high-precision 3D model including virtual skin, soft tissue and bones. It has the following functions:

[0180] AI automatic segmentation: Automatically segment target organs, lesions, blood vessels, nerves and other key anatomical structures based on deep learning algorithms (such as convolutional neural networks) to ensure segmentation accuracy and efficiency.

[0181] Multimodal data fusion: Through image registration and fusion technology, image data from different modalities (such as CT and CTA) are integrated to generate a unified and complete three-dimensional model of the surgical area.

[0182] Three-dimensional reconstruction: Generate high-resolution three-dimensional images to provide doctors with a three-dimensional and intuitive display of the surgical target area, facilitating further path planning and surgical implementation.

[0183] The ultrasound simulation and evaluation module is configured to simulate ultrasound imaging at different angles and positions according to the ultrasound image prediction method based on multimodal imaging data as described in any of the above embodiments, and to comprehensively evaluate the effects of the air-skin tissue interface and the soft tissue-bone interface on ultrasound imaging.

[0184] The ultrasound imaging prediction module is configured to generate a corresponding ultrasound prediction image based on the CT section determined by the ultrasound imaging requirement / exploration section.

[0185] The display module is configured to display three-dimensional models from multiple perspectives, display air-skin tissue interface evaluation results and soft tissue-bone interface evaluation results, the simulation process of ultrasonic imaging at different angles and positions, and predicted ultrasonic images and other information.

[0186] In summary, the ultrasound image prediction system based on multimodal image data utilizes the ultrasound image prediction method based on multimodal image data as described in any of the above embodiments to predict the corresponding ultrasound image under the target ultrasound imaging requirements according to the CT image of the target body.

[0187] In one embodiment of the present invention, a computer-readable storage medium is provided for storing program instructions, wherein the program instructions are configured to be called to execute the steps of the method described in any one of the above embodiments.

[0188] It should be noted that the above-mentioned ultrasonic image correction method based on multimodal image data, ultrasonic image prediction system based on multimodal image data, and computer-readable storage medium embodiments belong to the same inventive concept as the ultrasonic image prediction method embodiment based on multimodal image data, and the entire content of the ultrasonic image prediction method embodiment based on multimodal image data is incorporated into the ultrasonic image correction method based on multimodal image data, the ultrasonic image prediction system based on multimodal image data, and the computer-readable storage medium embodiments by reference.

[0189] It should be noted that, in this article, relational terms such as are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0190] The above is only a specific implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. An ultrasound image prediction method based on multimodal image data, characterized in that: The method comprises the following steps: Building a three-dimensional model of the target body based on multimodal image data, wherein the multimodal image data includes at least two of CT images, CTA images, and ultrasound images, and determining that a spatial coordinate system corresponding to the three-dimensional model is a first coordinate system; Acquire a CT image of the target body, determine that a spatial coordinate system corresponding to the CT image is a second coordinate system, and determine a mapping relationship from the first coordinate system to the second coordinate system; Based on the three-dimensional model, simulating an ultrasonic probe to explore the three-dimensional model, and determining a section of the three-dimensional model to be explored by the ultrasonic probe under target ultrasonic imaging requirements; Determine the position coordinates of the exploration section in the first coordinate system, and map the position coordinates of the exploration section to the second coordinate system according to a mapping relationship from the first coordinate system to the second coordinate system to obtain a CT section; The ultrasonic image corresponding to the exploration section is predicted according to the CT section.

2. The ultrasonic image prediction method based on multimodal image data according to claim 1, characterized in that: The target ultrasound imaging requirement is an optimal ultrasound imaging requirement, and the exploration section determined based on the optimal ultrasound imaging requirement is the optimal exploration section; The three-dimensional model includes virtual skin tissue, soft tissue and bones matching the target body, and the optimal exploration section is determined by: Simulating an ultrasonic probe to explore the three-dimensional model in different positions and postures to obtain a plurality of simulated ultrasonic images; Performing an evaluation operation on each simulated ultrasound imaging to obtain an evaluation result, wherein the evaluation operation includes air-skin tissue interface evaluation and / or soft tissue-bone interface evaluation; Determine the optimal evaluation result and its corresponding optimal simulated ultrasonic imaging from the evaluation results corresponding to the respective simulated ultrasonic imaging; The posture of the ultrasound probe corresponding to the optimal simulated ultrasound imaging is taken as the optimal posture, and the optimal exploration section corresponding to the ultrasound probe in the optimal posture is determined.

3. The ultrasonic image prediction method based on multimodal image data according to claim 2, characterized in that: The air-skin tissue interface assessment comprises the following steps: Determining a projection direction of the ultrasound probe relative to the three-dimensional model according to the position and posture of the ultrasound probe, and calculating a projection area of ​​the ultrasound probe on the surface of the skin tissue along the projection direction; Determining a surface area of ​​the skin tissue corresponding to the projection area as a projection surface, and performing meshing processing on the projection surface to generate a polygonal mesh; For each vertex on the polygonal mesh, determine a line segment from the vertex along the opposite direction of the projection direction to the surface of the ultrasound probe, and calculate the length of the line segment; The air-skin tissue interface was evaluated based on the length of all line segments.

4. The ultrasonic image prediction method based on multimodal image data according to claim 3, characterized in that: The air-skin tissue interface assessment result is determined based on the length of all line segments, including the following steps: Calculate the average and variance of the lengths of all line segments, and count the percentage of line ends whose lengths are less than the preset length to the total number of line ends; The air-skin tissue interface is evaluated according to the mean value, variance and percentage to obtain the air-skin tissue interface evaluation result.

5. The ultrasonic image prediction method based on multimodal image data according to claim 2, characterized in that: The soft tissue-bone interface assessment comprises the following steps: Determine the location of the ultrasonic sound source of the ultrasonic probe according to the position of the ultrasonic probe, and configure a fan-shaped area in the detection section of the ultrasonic probe with the ultrasonic sound source as the center of the circle; An estimated value of the bone occluding ultrasound in the fan-shaped area is calculated, and a soft tissue-bone interface assessment is performed based on the estimated value of the bone occluding ultrasound to obtain a soft tissue-bone interface assessment result.

6. The ultrasonic image prediction method based on multimodal image data according to claim 5, characterized in that: The estimated value of the bone occlusion ultrasound is calculated as follows: Arrange a plurality of rays starting from the center of the circle and extending toward the three-dimensional model in the fan-shaped region, wherein the plurality of rays are evenly distributed in the fan-shaped region; According to the principle of ultrasonic imaging, a weight value is configured for each of the rays based on Gaussian normal distribution; Determine the intersection line segment of each ray and the bone, and calculate the estimated value of the ray being blocked by the bone according to the weight value and the intersection line segment; The estimated value of the bone-occluded ultrasound is calculated based on the estimated values ​​of all rays being blocked by the bones.

7. The ultrasonic image prediction method based on multimodal image data according to claim 6, characterized in that: The calculation formula of the estimated value of the bone occlusion ultrasound is expressed as follows: Wherein, WAvg is the estimated value of the bone blocking ultrasound, m is the number of the rays, j represents a positive integer from 1 to m, Lj represents the total length of the intersection line segment between the jth ray and the bone, θ represents the angle between the th ray and the center line of the ultrasound probe, and σ represents the standard deviation.

8. The ultrasonic image prediction method based on multimodal image data according to claim 1, characterized in that: Predicting the ultrasound image corresponding to the exploration section according to the CT section includes the following steps: Inputting the data corresponding to the CT section into a pre-constructed ultrasound image prediction model to output a predicted ultrasound image corresponding to the exploratory section; The ultrasound image prediction model is established by the following steps: Collecting a learning sample set, each sample in the learning sample set includes a CT image and an ultrasound image of the same object; Designing a basic model, the basic model comprising an encoder, a decoder and a reconstruction loss calculation module, wherein the encoder is configured to extract image features of the CT image, the decoder is configured to generate a pseudo ultrasound image according to the image features, and the reconstruction loss calculation module is configured to calculate the reconstruction loss of the pseudo ultrasound image; The basic model is trained using a preset loss function so that the pseudo ultrasound image generated by the trained model based on the CT image is consistent with the real ultrasound image, thereby obtaining the ultrasound image prediction model.

9. The ultrasonic image prediction method based on multimodal image data according to claim 8, characterized in that: The reconstruction loss calculation module is configured to calculate the reconstruction loss of the pseudo ultrasound image according to the following formula: L Rec =||R.US-P.US|| 2 Among them, L Rec represents the reconstruction loss, R.US represents the real ultrasound image, and P.US represents the pseudo ultrasound image.

10. The ultrasonic image prediction method based on multimodal image data according to claim 9, characterized in that: It also includes a discriminator, which is configured to judge the authenticity of each area in the pseudo ultrasound image, and its calculation formula is: L Dis =log(Dis(R.US))+log(1-(Dis(P.US))) Among them, L Dis Represents the authenticity calculation result of the region, Dis(R.US) represents the output result of the region discriminator with the ultrasonic image as input, and Dis(P.US) represents the output result of the region discriminator with the pseudo ultrasonic image as input.

11. The ultrasonic image prediction method based on multimodal image data according to claim 8, characterized in that: The basic model further comprises a quantization module, and during the training process of the basic model, the quantization module is configured to perform quantization processing on the image features to obtain quantized encoding information; The decoder is configured to generate the pseudo ultrasound image according to the quantized encoding information.

12. The ultrasonic image prediction method based on multimodal image data according to claim 11, characterized in that: The calculation formula of the quantization loss is expressed as follows: Among them, L q represents the quantization loss, E c represents the image features, detach() represents the separation and replication of the gradient of the feature vector during model training, E q represents the quantized coding information.

13. The ultrasonic image prediction method based on multimodal image data according to claim 1, characterized in that: The three-dimensional model is constructed by the following steps: Acquiring multimodal image data of the target body, wherein the multimodal image data includes at least two of CT images, CTA images, and ultrasound images; Segmenting a target region in the multimodal image data based on a deep learning algorithm, wherein the target region includes one or more of a target organ, a lesion area, a blood vessel, a nerve, and a bone; The multimodal image data are registered and integrated through image registration and fusion technology to generate the three-dimensional model.

14. An ultrasonic image correction method based on multimodal image data, characterized in that: The following steps are involved: For a target object, under the target ultrasound imaging requirement, obtaining a real ultrasound image of the target object; Based on the ultrasonic image prediction method based on multimodal image data as described in any one of claims 1 to 13, predicting the corresponding predicted ultrasonic image under the target ultrasonic imaging requirements; The real ultrasound image is corrected using the predicted ultrasound image.

15. An ultrasound image prediction system based on multimodal image data, characterized in that: The ultrasonic image prediction method based on multimodal image data as described in any one of claims 1 to 13 is used to predict the corresponding ultrasonic image under the target ultrasonic imaging requirements according to the CT image of the target body.

16. A computer-readable storage medium for storing program instructions, characterized in that: The program instructions are configured to be called to execute the steps of the method according to any one of claims 1 to 13.

Images