A volume rendering fusion rendering method based on multi-modal ultrasound data

Abstract

The application belongs to the medical field and particularly relates to a volume rendering fusion rendering method based on multi-modal ultrasound data, which comprises the following steps: performing volume rendering on sound attenuation data, obtaining a first threshold range by windowing; extracting an isosurface polyhedron according to the first threshold range; cutting the volume data using the polyhedron to construct binary volume data; processing three-modal data using the binary volume data, performing volume rendering on the processed reflection data, obtaining a second threshold range by windowing to clearly show the internal and external tissue structure and morphology of the breast; and performing fusion volume rendering on the processed reflection data, sound velocity data and sound attenuation data to obtain a fusion image. The sound attenuation data is used as a basis to remove imaging impurities in the multi-modal data, thereby providing a data basis with less noise and impurities for volume rendering, and avoiding problems such as unclear tissue boundaries and impurity shielding in subsequent data fusion display.

Description

Technical Field

[0001] This invention belongs to the medical field, specifically relating to a volume rendering and fusion method based on multimodal ultrasound data. Background Technology

[0002] Volume rendering is a three-dimensional visualization method that is directly applied to three-dimensional volume data of medical images such as CT (computed tomography) or MRI (magnetic resonance imaging). The applicant has developed a device that uses ultrasound to acquire reflected and transmitted wave signals from different locations of breast tissue, and further obtains three modal medical imaging data of the same target space based on ultrasound reflection, sound velocity, and attenuation imaging.

[0003] However, volume rendering of individual modal data cannot fully realize the advantages of this device in diagnosis and treatment. Furthermore, if volume rendering images of different modalities are compared with each other, there will be problems such as obstruction of visual effects and difficulty in positioning.

[0004] This invention primarily utilizes data from three different modalities generated by the device, simultaneously serving as input for volume rendering, to achieve a three-modal fusion volume rendering method for 3D visualization. During the fusion volume rendering process, the data from the three modalities, based on their respective characteristics, influence the shape, color, transparency, and other effects of the final 3D image formed by the volume rendering. This allows the system to depict the overall shape according to settings while eliminating interference and impurities, highlighting specific types or regions of tissue, and helping users better explore the internal structure and tissue types of the breast. Summary of the Invention

[0005] To overcome the shortcomings of existing technologies, this invention provides a volume rendering and fusion method based on multimodal ultrasound data.

[0006] A volume rendering and fusion method based on multimodal ultrasound data includes:

[0007] Step 1: Use ray casting to perform volume rendering on the sound attenuation data, and obtain a first threshold range that can clearly display the outline of the entire breast tissue by adjusting the window.

[0008] Step 2: Extract the isosurface polyhedron based on the first threshold range obtained in Step 1;

[0009] Step 3: Use the polyhedral cut volume data to construct the binarized volume data;

[0010] Step 4: Remove noise. Use binary volume data to process the three-modal data, which includes reflection data, sound velocity data, and sound attenuation data. Use ray casting to perform volume rendering on the processed reflection data. Use windowing to obtain a second threshold range to clearly show the internal and external tissue structure and morphology of the breast.

[0011] Step 5: Perform a fusion rendering on the processed reflection data, sound velocity data, and sound attenuation data to obtain a fused image.

[0012] Further, step 2 includes:

[0013] Using the first threshold range obtained in step 1, the acoustic attenuation volume data is subjected to threshold segmentation algorithm and MarchingCubes algorithm to extract isosurfaces, segmenting a three-dimensional polyhedron model of breast tissue from the volume data. Then, the connectivity algorithm is used to decompose the model into multiple independent and unconnected polyhedron objects. Finally, the polyhedron volume calculation algorithm is used to remove the small-particle polyhedrons in the polyhedron model generated by isosurface extraction, leaving only the single polyhedron with the largest volume, that is, the overall contour polyhedron model of breast tissue.

[0014] Furthermore, step 3 includes:

[0015] The overall contour polygon model of the breast tissue obtained in step 2 is used to classify and binarize all points in the entire volume data. Points contained in the polygon model are assigned a value of 1, and the rest are assigned a value of 0. This binarized volume data is then saved.

[0016] Furthermore, step 4 includes:

[0017] The binary volume data obtained in step 3 is used to perform point-to-point multiplication with the reflection data, sound velocity data, and sound attenuation data to zero out the reflection signal values ​​of other areas outside the breast tissue in the original data. Then, the ultrasound reflection data is used as the volume rendering base data to perform volume rendering imaging, and a second threshold range is determined to clearly show the internal and external tissue structure and morphology of the breast.

[0018] Furthermore, in step 5, ray casting is used to fuse the reflection data, sound velocity data, and sound attenuation data into a composite image. The basic color and transparency calculation formulas for ray casting are as follows:

[0019]

[0020] Among them, C i With O iHere, represents the color and opacity of the incident light ray along the sampling point i, respectively; J is the decision function, calculated based on the data value v of the reflection data, returning 1 if v is within the second threshold range, otherwise returning 0; f is the function that correlates the data value v of the reflection data with color, and N is the function that correlates the data value v of the reflection data with opacity; f' is the function that correlates the sound velocity value s, the sound attenuation value a, and color, and the color calculation result of this function can be the R, G, and B values ​​in the RGB color space; N' is the function that calculates the sound velocity value s, the sound attenuation value a, and opacity, and the result is jointly determined by the sound velocity value s and the sound attenuation value a.

[0021] Compared with the prior art, the beneficial effects of the present invention are:

[0022] 1) Using acoustic attenuation data as a basis removes imaging impurities from multiple modal data, providing a data foundation with less noise and fewer impurities for volume rendering, and avoiding problems such as unclear tissue boundaries and impurity occlusion for subsequent data fusion and display;

[0023] 2) Compared with rendering images from individual data points, fusion display of multiple data points can achieve clearer, more explicit, and more intuitive visual effects and spatial perception effects;

[0024] 3) The shared reality of multimodal data, using color and transparency variations to showcase material properties, allows users to obtain a clearer, more complete, and accurate description of the target, fully leveraging the capabilities of ultrasound equipment that can generate three-modal data at once. Attached Figure Description

[0025] Figure 1 This is a flowchart of a volume rendering and fusion method based on multimodal ultrasound data according to the present invention.

[0026] Figure 2 The image shows the effect obtained by adjusting the window after drawing the acoustic attenuation data of the phantom in Example 1;

[0027] Figure 3 A rendering of the untrimmed reflective data of the phantom in Example 1;

[0028] Figure 4 This is a rendering of the model in Example 1, based on the reflection data of the polyhedron after cutting.

[0029] Figure 5 This is a rendering of the fusion effect of the phantom detection data in Example 1. Detailed Implementation

[0030] The invention will now be further described with reference to the accompanying drawings.

[0031] Please see Figure 1A volume rendering and fusion method based on multimodal ultrasound data, comprising:

[0032] Step 1: Use conventional ray-casting to perform volumetric imaging on the acoustic attenuation data to determine a first threshold range that can clearly display the entire breast tissue contour.

[0033] Step 2: Using the first threshold range obtained in Step 1, the acoustic attenuation volume data is subjected to threshold segmentation algorithm and MarchingCubes algorithm to extract isosurfaces, and a three-dimensional spatial polygonal model of breast tissue is segmented from the volume data. Then, the connectivity algorithm is used to decompose the model into multiple independent and unconnected polyhedral objects. Finally, the polyhedral volume calculation algorithm is used to remove the small-particle polyhedrons in the polygonal model generated by isosurface extraction, leaving only the single polyhedron with the largest volume, that is, the overall contour polygonal model of breast tissue.

[0034] Step 3: Use the polygon model obtained in Step 2 to classify and binarize all points in the entire volume data. Assign a value of 1 to the points within the polygon model and 0 to the rest. Save this binarized volume data, which is a binarized matrix.

[0035] Step 4: Perform point-to-point multiplication using the binarized volume data obtained in Step 3, along with the reflection data, sound velocity data, and sound attenuation data, to zero out the reflection signal values ​​of other areas outside the breast tissue in the original data. Then, using the ultrasound reflection data as the base data for volume rendering, perform volume rendering imaging using a standard ray projection method, and determine a second threshold range to clearly display the internal and external tissue structures and morphology of the breast.

[0036] Step 5: Using the modified ray casting method, the processed reflection data, sound velocity data, and sound attenuation data are used to draw the fused body, resulting in a fused image.

[0037] Ray-casting is a well-known technique. Its core idea is to emit a ray of light from each pixel on the screen along the line of sight. As this ray passes through the volume data, samples are taken at equal intervals along the ray's direction. Interpolation is then used to calculate the color value and opacity of the sampled points. During the sampling process, the light intensity gradually decreases according to a composite lighting model of light absorption and divergence, based on the opacity and color of the current sampled point. When the light intensity decreases to 0, sampling stops. Then, the sampled points on the ray are composited in a front-to-back or back-to-front order to calculate the color value of the corresponding pixel on the screen, thus completing the volume rendering of the object from the current viewpoint.

[0038] This invention is also a volume rendering display. Therefore, it uses the ray casting method as a basis and makes certain modifications to achieve volume rendering imaging of three-dimensional volume data. The color synthesis formula and image synthesis operation are consistent with the original method. The difference lies in the introduction and calculation process of two other modal data in the color and transparency formulas.

[0039] Formulas related to ray projection:

[0040] (1) Formula for color synthesis at each sampling point along the ray in the ray projection method:

[0041]

[0042] Where C is the final color of the screen pixel, C i With O i Let i represent the color and opacity of the incident light along the sampling point i, and C represent the superposition of the color and opacity at all sampling points along the path.

[0043] (2) From formula (1), the recursive formula for superimposing colors from front to back sampling points can be simplified and derived:

[0044]

[0045] Among them, C' i The color is a weighted cumulative color based on the transparency from the first sampling point to the i-th sampling point, O' i To accumulate opacity, when O' i When the value increases to 1, the ray will not be sampled again.

[0046] (3) Formulas for color and transparency at each sampling point:

[0047]

[0048] Where v is the volume data value corresponding to the sampling point, f is the correlation calculation function between color and volume data value, and N is the correlation calculation function between opacity and volume data value. The calculation of the data value v at the location of the sampling point and the actual data values ​​of the volume data around that sampling point can be performed using conventional methods such as the nearest distance method and linear interpolation, which will not be elaborated here.

[0049] Furthermore, since the three-modal data designed in this invention is generated by a single scan and consists of different data collected from the same space, spatial registration is not required.

[0050] During the volume rendering of the reflection data in step 1, the existing volume rendering object is given a base transparency and base color using the functions f(v) and N(v). This base color is generally a gray with consistent three-channel digital values ​​in the RGB color space, so as to distinguish it from the colors that will be blended later.

[0051] After determining the second reflection threshold, two additional modal data points, sound velocity and sound attenuation, are introduced and incorporated into the calculation of base color and transparency in the volume data. That is, when calculating color and transparency, the relevant thresholds, coefficients, and formulas for sound velocity and sound attenuation are used to calculate the color and transparency of the sampling points.

[0052] The formula is as follows:

[0053]

[0054] The above formula is an improvement on formula (3). J is a judgment function, which is calculated based on the data value v of the reflection data. If v is within the threshold 1, it returns 1; otherwise, it returns 0. f' is a function that calculates the correlation between the sound velocity value s, the sound attenuation value a, and the color. The color calculation result of this function can be the R, G, and B values ​​in the RGB color space. Generally, the sound velocity value will affect the value change of a single channel, such as the R channel, while the sound attenuation will affect the value change of another color channel, such as the G channel. N' is a function that calculates the sound velocity value, the sound attenuation value, and the opacity. The result is determined by the sound velocity value and the sound attenuation value. The color synthesis formula is consistent with the original formula of the algorithm.

[0055] Example 1

[0056] The implementation of this invention utilizes the Visualization Toolkit (VTK) library for coding the visualization program. VTK is an open-source visualization library, considered one of the best open-source visualization libraries for 3D computer graphics, image processing, and visualization. It includes basic implementations of the RayCasting algorithm in multiple versions, including CPU and GPU versions, and allows for convenient adjustments and modifications. The following is a rendering program developed using a modified VTK volume rendering algorithm based on the principles described in this paper, demonstrating the effect of volume rendering and fusion rendering on a set of experimental data.

[0057] like Figure 2 As shown, firstly, the acoustic attenuation data volume is plotted and windowed to obtain a threshold range with a clear target boundary.

[0058] Then, isosurfaces are extracted using this threshold range to generate a polyhedron, which is then used to clip the data from the three modalities. The clipped reflection data is then used for volume rendering, and windowing is adjusted to achieve a clear display effect with distinct internal and external structures. For example... Figure 3As shown, before cropping, the imaging is affected by the reflected signals from the surrounding tissue, resulting in poor transparency and unclear internal structures. Figure 4 As shown, after cropping, the influence of reflection signals around the tissue is eliminated, the transparency is good, and the internal structure is clearly displayed, but due to the transparency, the three-dimensional effect is not strong.

[0059] Next, the logic of the color calculation function and transparency calculation function was rewritten through encoding. Sound velocity and sound attenuation data were introduced, and specific threshold ranges were set for each data point. Then, a three-modal data volume rendering and integration process was performed, resulting in the following effect: Figure 5 As shown.

[0060] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A volume rendering and fusion method based on multimodal ultrasound data, characterized in that, include: Step 1: Use ray casting to perform volume rendering on the sound attenuation data, and obtain a first threshold range that can clearly display the outline of the entire breast tissue by adjusting the window. Step 2: Extract the isosurface polyhedron based on the first threshold range obtained in Step 1; Step 3: Use the polyhedral cut volume data to construct the binarized volume data; Step 4: Remove noise. Use binary volume data to process the three-modal data, which includes reflection data, sound velocity data, and sound attenuation data. Use ray casting to perform volume rendering on the processed reflection data. Use windowing to obtain a second threshold range to clearly show the internal and external tissue structure and morphology of the breast. Step 5: Perform a fusion rendering on the processed reflection data, sound velocity data, and sound attenuation data to obtain a fused image. The fusion rendering is performed using ray casting. The basic color and transparency calculation formulas for ray casting are as follows: ， Among them, C i With O i Here, represents the color and opacity of the incident light ray along the sampling point i, respectively; J is the decision function, calculated based on the data value v of the reflection data, returning 1 if v is within the second threshold range, otherwise returning 0; f is the function that correlates the data value v of the reflection data with color, and N is the function that correlates the data value v of the reflection data with opacity; f' is the function that correlates the sound velocity value s, the sound attenuation value a, and color, and the color calculation result of this function can be the R, G, and B values ​​in the RGB color space; N' is the function that calculates the sound velocity value s, the sound attenuation value a, and opacity, and the result is jointly determined by the sound velocity value s and the sound attenuation value a.

2. The volume rendering and fusion method based on multimodal ultrasound data according to claim 1, characterized in that, Step 2 includes: Using the first threshold range obtained in step 1, the acoustic attenuation volume data is subjected to threshold segmentation algorithm and MarchingCubes algorithm to extract isosurfaces, segmenting a three-dimensional polyhedron model of breast tissue from the volume data. Then, the connectivity algorithm is used to decompose the model into multiple independent and unconnected polyhedron objects. Finally, the polyhedron volume calculation algorithm is used to remove the small-particle polyhedrons in the polyhedron model generated by isosurface extraction, leaving only the single polyhedron with the largest volume, that is, the overall contour polyhedron model of breast tissue.

3. The volume rendering and fusion method based on multimodal ultrasound data according to claim 2, characterized in that, Step 3 includes: The overall contour polygon model of the breast tissue obtained in step 2 is used to classify and binarize all points in the entire volume data. Points contained in the polygon model are assigned a value of 1, and the rest are assigned a value of 0. This binarized volume data is then saved.

4. The volume rendering and fusion method based on multimodal ultrasound data according to claim 1, characterized in that, Step 4 includes: The binary volume data obtained in step 3 is used to perform point-to-point multiplication with the reflection data, sound velocity data, and sound attenuation data to zero out the reflection signal values ​​of other areas outside the breast tissue in the original data. Then, the ultrasound reflection data is used as the volume rendering base data to perform volume rendering imaging, and a second threshold range is determined to clearly show the internal and external tissue structure and morphology of the breast.

Images