Medical image rendering method, electronic equipment and computer readable storage medium

Abstract

The embodiment of the invention provides a medical image rendering method, electronic equipment and a computer readable storage medium. The method comprises the following steps: acquiring first volume data and surface grid data; performing semitransparent rendering on the surface grid data to obtain a first rendering result including information of each grid piece element; volume rendering is carried out on second volume data according to the first rendering result, a second rendering result is obtained, the second volume data is formed by the first volume data and a first grid piece element, and the first grid piece element is a grid piece element located in the first volume data; and according to the first rendering result, performing color mixing operation on a second grid piece element and a second rendering result to obtain a medical image, the second grid piece element being a grid piece element located outside the first volume data. According to the embodiment of the invention, mixed rendering is carried out on the surface grid data and the volume data in combination with volume rendering and semitransparent rendering, so that the semitransparent biological tissue or the medical device can be displayed in the biological tissue obtained by volume rendering.

Description

Technical Field

[0001] The present application belongs to the field of medical imaging technology, and in particular, relates to a medical image rendering method, an electronic device, and a computer-readable storage medium. Background Art

[0002] In the field of medical imaging, volume rendering is used to render volume data, so that the biological tissue (such as the liver, etc.) corresponding to the volume data in the rendered medical image can be made more realistic; semi-transparent rendering is used to render surface mesh data, so that the biological tissue or medical device corresponding to the surface mesh data in the rendered medical image can be displayed semi-transparently.

[0003] At present, when rendering volume data or surface mesh data separately, a rendering method such as volume rendering or semi-transparent rendering can be used accordingly to obtain a medical image with a corresponding effect. However, when performing mixed rendering of volume data and surface mesh data, it is currently not possible to combine volume rendering and semi-transparent rendering for mixed rendering, and thus it is not possible to display semi-transparent biological tissues or medical devices (obtained by semi-transparent rendering of surface mesh data) in biological tissues obtained by volume rendering. Summary of the invention

[0004] The embodiments of the present application provide a medical image rendering method, an electronic device, and a computer-readable storage medium, which can display semi-transparent biological tissue or medical equipment in biological tissue obtained by volume rendering.

[0005] In a first aspect, an embodiment of the present application provides a medical image rendering method, comprising:

[0006] Acquire first volume data of a first object and surface mesh data of a second object;

[0007] Performing semi-transparent rendering on the surface mesh data to obtain a first rendering result, wherein the first rendering result includes information of each mesh fragment;

[0008] Performing volume rendering on second volume data according to the first rendering result to obtain a second rendering result, wherein the second volume data is volume data formed by the first volume data and a first mesh fragment, and the first mesh fragment is a mesh fragment located inside the first volume data;

[0009] According to the first rendering result, a color blending operation is performed on the second grid fragment and the second rendering result to obtain a medical image, where the second grid fragment is a grid fragment located outside the first volume data.

[0010] The embodiment of the present application first performs semi-transparent rendering on the surface mesh data to obtain a first rendering result; then the first mesh fragment located inside the first volume data is added to the first volume data as a new medium to form a new second volume data, and the newly formed second volume data is volume rendered to obtain a second rendering result; finally, the second mesh fragment located outside the first volume data is mixed with the second rendering result, thereby realizing mixed rendering of the surface mesh data and the volume data based on volume rendering and semi-transparent rendering, and then semi-transparent biological tissue or medical device can be displayed in the biological tissue obtained by volume rendering. That is, in the rendered medical image, the biological tissue or medical device corresponding to the surface mesh data is displayed semi-transparently, and through the semi-transparent biological tissue or medical device, the biological tissue obtained by volume rendering of the volume data can be observed.

[0011] In some possible implementations of the first aspect, performing volume rendering on the second volume data according to the first rendering result to obtain the second rendering result includes:

[0012] Construct a ray and project it to the second volume data, so that the ray passes through the screen pixels;

[0013] If a grid fragment is drawn on the screen pixel, mark the ray;

[0014] Iterate distance sampling of the ray;

[0015] For each distance sampling, if the light is marked, determine whether the light passes through the grid medium between the first time point and the second time point according to the first rendering result; if so, perform a color mixing operation on the grid medium passed by the light to obtain a grid color mixing result; determine the color contribution value of the sampling point according to the medium type of the sampling point obtained by the distance sampling and the grid color mixing result, and perform directional sampling according to the medium type of the sampling point;

[0016] For each distance sampling, if the light does not pass through the grid medium between the first time point and the second time point, or the light is not marked, the color contribution value of the sampling point is determined according to the medium type of the sampling point obtained by the distance sampling, and the direction sampling is performed according to the medium type of the sampling point;

[0017] After the light distance sampling is completed, the color information of the screen pixel is obtained according to the color contribution value of each sampling point;

[0018] Among them, the second volume data includes grid medium, real medium and mixed medium, the grid medium is the first grid fragment, the real medium is the first volume data, the mixed medium is a mixture of the grid medium and the real medium, the first time point is the initial time of the distance sampling, the second time point is determined based on the first time point and the sampling step, and the second rendering result includes the color information of the screen pixel points.

[0019] In this implementation, the mesh fragment located inside the volume data is added to the volume data as a new medium, so as to add the self-luminous medium (i.e., the first mesh fragment) to the volume data that originally does not contain the self-luminous medium, and ray tracing is performed based on the added new medium and the original volume data; during the ray tracing process, the mesh medium through which the light passes is mixed to achieve a semi-transparent display effect of the pixel points. In this way, the result of semi-transparent rendering is effectively used as the self-luminous medium to participate in the solution of the rendering equation in the volume rendering process, thereby achieving mixed rendering of volume data and surface mesh data.

[0020] In some possible implementations of the first aspect, determining, according to the first rendering result, whether the light passes through the grid medium between the first time point and the second time point includes:

[0021] Determine a first position reached by the light at a first time point, and a second position reached by the light at a second time point;

[0022] determining a first depth at a first location and a second depth at a second location;

[0023] Projecting the first position and the second position onto a screen to obtain a first screen point on the screen at the first position and a second screen point on the screen at the second position;

[0024] When the first screen point and the second screen point coincide with each other, a first target grid fragment is acquired according to information of each grid fragment; if the first target grid fragment is acquired, it is determined that the light passes through the grid medium; if the first target grid fragment is not acquired, it is determined that the light does not pass through the grid medium; the first target grid fragment is a grid fragment located on the first screen point and having a depth between the first depth and the second depth;

[0025] When the first screen point and the second screen point do not coincide with each other, for each pixel sampling point on the line connecting the first screen point and the second screen point, according to information of each grid fragment, obtaining a candidate grid fragment located on the pixel sampling point and having a depth between the first depth and the second depth, and determining a distance between each candidate grid fragment and the light;

[0026] If the second target grid fragment exists, it is determined that the light passes through the grid medium; if the second target grid fragment does not exist, it is determined that the light does not pass through the grid medium, and the second target grid fragment is a candidate grid fragment whose distance is less than or equal to the preset distance threshold.

[0027] Compared with the method of solving the intersection point of light and mesh fragments through mathematical calculation, this implementation method determines the intersection point of light and mesh fragments through the first rendering result, which has less calculation amount and faster calculation speed.

[0028] In some possible implementations of the first aspect, performing a color mixing operation on a grid medium through which light passes to obtain a grid color mixing result includes:

[0029] For each target mesh fragment after depth sorting, the mixed opacity of the current depth level is obtained according to the opacity of the target mesh fragment in the current depth level and the mixed opacity of the previous depth level; the mixed color of the current depth level is obtained according to the color and opacity of the target mesh fragment in the current depth level and the mixed color and mixed transparency of the previous depth level; the depth of the current depth level is greater than the depth of the previous depth level;

[0030] The grid blending result includes the blended color and blended opacity of the depth level with the greatest depth;

[0031] The grid medium through which the light passes includes at least one target grid fragment.

[0032] In this implementation, during the ray tracing process, the grid medium through which the light passes is mixed in sequence along the direction of increasing depth to achieve a semi-transparent display effect of the pixel points.

[0033] In some possible implementations of the first aspect, determining the color contribution value of the sampling point according to the medium type of the sampling point obtained by distance sampling and the grid color mixing result includes:

[0034] If the sampling point is a real medium, determine the direct illumination information of the sampling point; determine the color contribution value of the sampling point based on the direct illumination information and the grid color mixing result;

[0035] If the sampling point is a grid medium, determine the self-luminous information of the sampling point according to the grid color mixing result and the color information of the third target grid fragment corresponding to the sampling point; determine the color contribution value of the sampling point according to the self-luminous information;

[0036] If the sampling point is a mixed medium, determine the direct illumination information of the sampling point, and determine the self-luminous information of the sampling point based on the grid color mixing result and the color information of the third target grid fragment corresponding to the sampling point; determine the color contribution value of the sampling point based on the self-luminous information and the direct illumination information.

[0037] In this implementation, the result of semi-transparent rendering is effectively used to participate in solving the rendering equation of the volume rendering process.

[0038] In some possible implementations of the first aspect, determining the color contribution value of the sampling point according to the medium type of the sampling point obtained by distance sampling includes:

[0039] If the sampling point is a real medium, determine the direct illumination information of the sampling point; and determine the color contribution value of the sampling point based on the direct illumination information;

[0040] If the sampling point is a grid medium, determine the self-luminous information of the sampling point according to the color information of the third target grid element corresponding to the sampling point; determine the color contribution value of the sampling point according to the self-luminous information;

[0041] If the sampling point is a mixed medium, determine the direct illumination information of the sampling point, and determine the self-luminous information of the sampling point according to the color information of the third target grid element corresponding to the sampling point; determine the color contribution value of the sampling point according to the self-luminous information and the direct illumination information.

[0042] In this implementation, the result of semi-transparent rendering is effectively used to participate in solving the rendering equation of the volume rendering process.

[0043] In some possible implementations of the first aspect, after obtaining the sampling point and before determining the color contribution value of the sampling point, the method further includes:

[0044] Determine the third depth of the sampling point and the target screen pixel corresponding to the sampling point;

[0045] According to the information of each grid fragment, a third target grid fragment is obtained, where the third target grid fragment is located on the target screen pixel point and the difference between the depth and the third depth is less than or equal to the preset depth threshold;

[0046] If the opacity of the third target grid element is zero, it is determined that the sampling point is a grid medium;

[0047] If the opacity of the third target grid fragment is not zero, it is determined that the sampling point is a mixed medium.

[0048] In some possible implementations of the first aspect, directional sampling is performed according to a medium type at a sampling point, including:

[0049] If the sampling point is a real medium or a mixed medium, the scattering mode of the sampling point is determined according to the normal vector of the real medium. The scattering mode of the sampling point is volume scattering or surface scattering.

[0050] If the sampling point is a grid medium, the propagation direction of the light is maintained.

[0051] In some possible implementations of the first aspect, performing semi-transparent rendering on the surface mesh data to obtain a first rendering result includes:

[0052] Performing semi-transparent rendering on the surface mesh data to obtain a first linked list and a first texture image, wherein the first linked list includes color information and depth information of each mesh fragment, a flag bit indicating whether the mesh fragment is inside the first volume data, and index information indicating the position of the mesh fragment last drawn on the screen pixel point in the linked list, and the first texture image is used to indicate the position of the mesh fragment last drawn on the screen pixel point in the linked list;

[0053] For each screen pixel, each grid fragment on the screen pixel is converted to the coordinate system of the first volume data to determine whether the grid fragment is located inside the first volume data; if the grid fragment is located inside the first volume data, the flag bit of the grid fragment is set to a first state; if the grid fragment is located outside the second volume data, the flag bit of the grid fragment is set to a second state;

[0054] For each screen pixel, the grid fragments on the screen pixel are depth-sorted according to the depth information of the grid fragments to obtain a depth sorting result;

[0055] According to the depth sorting result, the index information of the first texture image and the mesh fragments in the first linked list is modified to obtain the second texture image and the second linked list.

[0056] In some possible implementations of the first aspect, if a grid fragment is drawn on a screen pixel, marking the light includes:

[0057] If the target pixel point in the second texture image stores the position information of the grid fragment, the light is marked, and the target pixel point is the pixel point in the second texture image corresponding to the screen pixel point through which the light passes.

[0058] In some possible implementations of the first aspect, the second rendering result includes color information of the volume data point on the screen pixel point, and the color information of the volume data includes the color of the volume data point and the opacity of the volume data point;

[0059] According to the first rendering result, a color mixing operation is performed on the second grid fragment and the second rendering result to obtain a medical image, including:

[0060] Depth-sorting the volume data points and the second grid fragments located under the same screen pixel;

[0061] For each depth level after depth sorting, the mixed opacity of the current depth level is obtained according to the opacity of the target object in the current depth level and the mixed opacity of the previous depth level; if the target object in the current depth level is a volume data point, the mixed color of the current depth level is obtained according to the color of the target object in the current depth level and the mixed transparency of the previous depth level; if the target object in the current depth level is a second grid fragment, the mixed color of the current depth level is obtained according to the color and opacity of the target object in the current depth level, and the mixed color and mixed transparency of the previous depth level; the depth of the current depth level is greater than the depth of the previous depth level;

[0062] Output a medical image based on the blended color and blended opacity of the depth level with the greatest depth.

[0063] In this implementation, the semi-transparent rendering result is divided into two parts: the inside of the volume data and the outside of the volume data, and the two parts are mixed with the volume rendering result in different ways, thereby realizing mixed rendering of volume data and surface data.

[0064] In a second aspect, an embodiment of the present application provides a medical image rendering device, comprising:

[0065] An acquisition module, used to acquire first volume data of a first object and surface mesh data of a second object;

[0066] A semi-transparent mesh rendering module, used for performing semi-transparent rendering on the surface mesh data to obtain a first rendering result, wherein the first rendering result includes information of each mesh fragment;

[0067] A volume rendering module, configured to perform volume rendering on second volume data according to the first rendering result to obtain a second rendering result, wherein the second volume data is volume data formed by the first volume data and a first mesh fragment, and the first mesh fragment is a mesh fragment located inside the first volume data;

[0068] The color blending module is used to perform a color blending operation on the second grid fragment and the second rendering result according to the first rendering result to obtain a medical image, wherein the second grid fragment is a grid fragment located outside the first volume data.

[0069] In a third aspect, an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements a method as described in any one of the first aspects above when executing the computer program.

[0070] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in any one of the first aspects above is implemented.

[0071] In a fifth aspect, an embodiment of the present application provides a computer program product. When the computer program product is run on an electronic device, the electronic device executes any one of the methods described in the first aspect.

[0072] It can be understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] Figure 1 A schematic diagram of a ray tracing principle model provided in an embodiment of the present application;

[0074] Figure 2 A schematic diagram of the propagation path of light passing through volume data provided in an embodiment of the present application;

[0075] Figure 3 A schematic flowchart of a medical image rendering method provided in an embodiment of the present application;

[0076] Figure 4 A schematic diagram of a propagation path of light passing through new volume data provided in an embodiment of the present application;

[0077] Figure 5 A schematic diagram of the depth sorting of the grid media provided in the embodiment of the present application;

[0078] Figure 6 A schematic diagram of a second grid fragment and a color mixing process of a rendering result provided in an embodiment of the present application;

[0079] Figure 7 A schematic diagram of depth sorting of second grid fragments and volume data points provided in an embodiment of the present application;

[0080] Figure 8 A schematic block diagram of a semi-transparent rendering process of surface mesh data provided in an embodiment of the present application;

[0081] Fig. 9 The chain diagram provided for the embodiment of the present application is shown;

[0082] Fig.10 A schematic diagram of a head image provided in an embodiment of the present application;

[0083] Fig.11 A schematic diagram of a new head image provided in an embodiment of the present application;

[0084] Fig.12A schematic block diagram of a volume rendering process based on ray tracing provided in an embodiment of the present application;

[0085] Fig.13 A schematic block diagram of the process of finding the intersection of light and a grid medium provided in an embodiment of the present application;

[0086] Fig.14A A schematic diagram of screen coordinate points provided in an embodiment of the present application;

[0087] Fig. 14B A schematic diagram of a hybrid rendering effect provided in an embodiment of the present application;

[0088] Fig.15 A schematic block diagram of the structure of a medical image rendering device provided in an embodiment of the present application;

[0089] Fig.16 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0090] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures and technologies are provided to facilitate a thorough understanding of the embodiments of the present application.

[0091] In the field of medical imaging, volume data refers to data obtained by scanning the object with a scanning device and then performing three-dimensional reconstruction based on the scan data.

[0092] The scanning device may be, for example, a computed tomography (CT), magnetic resonance (MR), positron emission tomography (PET), ultrasonic imaging equipment, or X-ray machine and other medical imaging equipment.

[0093] The detection object can be a biological body such as a human body or an animal body; it can be a tissue of a biological body, such as vascular tissue, bone tissue or soft tissue, etc.; it can also be a sub-tissue within the tissue, such as a bronchus, a lung lobe or a blood vessel, etc.

[0094] The volume data may be volume data including at least one biological tissue. The volume data is usually composed of a number of voxels.

[0095] For example, the volume data of the vascular tissue of the human body is obtained by scanning and reconstructing the vascular tissue of the human body, and the volume data of the foot is obtained by performing CT scanning and reconstructing the foot of the human body.

[0096] Surface mesh data refers to data obtained by meshing the surface of a three-dimensional model of an object such as a biological tissue or a medical device, that is, the three-dimensional object is represented by a surface mesh (e.g., basic primitives such as triangular facets).

[0097] The surface mesh data and the volume data are both three-dimensional data. Image rendering is performed on the surface mesh data and the volume data to obtain a rendered image that can be displayed on a screen.

[0098] In the related art, surface mesh data and volume data are rendered using different rendering methods. For example, a physically-based volume rendering (PBVR) method is used to render volume data, and a semi-transparent rendering method is used to render surface mesh data. The semi-transparent rendering method may be, but is not limited to, an order-independent semi-transparent mesh rendering algorithm (Per-Pixel Linked List, PPLL).

[0099] Among them, PBVR is based on ray tracing for image rendering. Ray tracing is based on the propagation law of light in the medium. According to the principle of reversibility of light path, by tracing the light emitted from the virtual camera (or viewpoint), the process of light propagation in the three-dimensional scene is calculated to render the rendered image and obtain a more realistic rendering effect.

[0100] For example, see Figure 1 A schematic diagram of a ray tracing principle model provided in an embodiment of the present application is shown. The ray tracing model may include a virtual camera (also called a viewpoint or observation point), a projection screen (also called a screen or image plane), and scene objects and light sources set in a three-dimensional virtual scene. Figure 1 The scene object in this example is a sphere, but this is just an example.

[0101] Under normal circumstances, after the light source emits light, the light undergoes multiple refractions, reflections, or scattering processes in the three-dimensional scene and enters the virtual camera, allowing the virtual camera to capture relevant information in the three-dimensional scene. However, the light source emits light in all directions, and not all light will eventually enter the virtual camera. Therefore, if you track the light emitted from the light source, tracing the light that does not eventually enter the virtual camera is an invalid calculation. In order to reduce invalid calculations and improve calculation accuracy, according to the principle of reversibility of the light path, the light is traced in reverse, that is, a light is emitted from the virtual camera, the light passes through the pixels on the projection screen, propagates in the three-dimensional scene, and finally returns to the light source.

[0102] like Figure 1As shown in the figure, a ray is emitted from a virtual camera to a three-dimensional scene, and the propagation process of the ray is traced. The ray passes through the pixels on the projection screen. It is detected whether the ray intersects with the scene objects. If the intersection operation determines that the ray intersects with the scene objects, the position of the intersection point is determined, and the direction of the new ray emitted from the intersection point is calculated; if the new ray continues to intersect with the scene objects, the ray tracing algorithm is recursively applied to simulate the propagation and interaction of the ray. Finally, according to the illumination calculation and other processes, the color of the pixel point through which the ray passes is determined, and the rendered image is obtained according to the color of the pixel point.

[0103] When volume rendering is performed on volume data based on ray tracing, a rendered image can be obtained by tracing the propagation process of light within the volume data.

[0104] Volume data is divided into real media and virtual media. The division rule between virtual media and real media is calculated based on the opacity of the media. If the light can completely pass through the medium, then the medium is virtual media, otherwise it is real media. In other words, virtual media will not change the propagation direction of light and will not participate in rendering. Therefore, when light propagates in volume data, the propagation direction of light does not change when it encounters virtual media, but volume scattering or surface scattering will occur when it encounters real media.

[0105] For example, see Figure 2 The schematic diagram of the propagation path of light through volume data provided by the embodiment of the present application is shown. After the light emitted from the virtual camera passes through the pixel points on the screen, it propagates in the volume data. The volume data includes a medium (i.e. Figure 2 ). Among them, Figure 2 In the light propagation path shown, the light passes through medium 1, medium 2, medium 3 and medium 4 in sequence. Medium 1 to medium 4 are all real media, and the light undergoes surface scattering in medium 1 and medium 4, and volume scattering in medium 2 and medium 3.

[0106] In the process of light propagation in volume data, multiple sampling points can be obtained by sampling the distance of the light. For each sampling point, the rendering equation is solved according to the direct illumination information and self-luminous information of the sampling point, and then the color value contribution of the sampling point is obtained; finally, the color information of the screen pixel point through which the light passes is obtained according to the color value contribution of multiple sampling points.

[0107] For example, the rendering equation may be as shown in Formula 1 below:

[0108]

[0109] Among them, L(x,ω) is the illumination energy of the medium at position x; position z represents the outside of the volume data; a certain position of the background is at a distance z from position x.

[0110] Transmittance: represents the probability of a photon successfully passing from position x1 to position x2, or the ratio of the outgoing light to the incident light from position x1 to position x2. L0(z,w) represents the energy of the background light, L e represents the energy of spontaneous light in a certain medium, Ls represents the energy of scattered light in a certain medium, μ a represents the light absorption coefficient, μ s represents the light scattering coefficient, μ t represents the extinction coefficient, and ω represents the direction of light.

[0111] In specific applications, the Monte Carlo method can be used to analyze the above formula 1, and then the simplified formula of the following formula 2 can be obtained:

[0112]

[0113] Where L(x0,ω0) is the light intensity of the sampling point x0 in the direction ω0; in the process of calculating the light intensity of the sampling point x0, the light contribution of other sampling points to the sampling point is calculated in an iterative manner. The light contribution for each sampling point includes the self-illumination term and the direct light term. Where ρ represents the albedo (a physical property of the medium), L d Indicates direct lighting.

[0114]

[0115] Among them, f p It indicates the distribution of the direction of light after passing through a certain medium. It can be generally expressed by bidirectional reflectance distribution function (BRDF) or phase function. BRDF and phase function are functions used to describe the propagation and reflection behavior of light in materials or media. p represents the probability density of a certain sampling point.

[0116] As can be seen from the above rendering equation, the rendering equation can be solved by calculating the direct illumination information and self-luminous information of the sampling point, and then the color contribution value of the sampling point can be calculated. When the volume data is sampled at a distance, the virtual medium does not participate in the rendering, that is, only the real medium is sampled, and the medium of the volume data itself does not emit light. Therefore, when solving the rendering equation, there is no need to consider the self-luminous term of the real medium, that is, there is no need to calculate the self-luminous information.

[0117] Normally, volume rendering is used to display volume data, and semi-transparent rendering is used to display surface meshes. The two rendering methods are independent of each other. However, in practical applications, it is necessary to perform mixed rendering of surface mesh data and volume data in the same scene. However, in related technologies, volume rendering algorithms do not support semi-transparent rendering algorithms, and thus cannot combine volume rendering and semi-transparent rendering to achieve mixed rendering of volume data and surface data. Ultimately, semi-transparent biological tissues or medical devices (obtained by semi-transparent rendering of surface mesh data) cannot be displayed in biological tissues obtained by volume rendering.

[0118] In response to the related technical issues mentioned above, an embodiment of the present application provides a medical image rendering solution. In this solution, the rendering result of the semi-transparent rendering is divided into two parts: the grid fragments located in the volume data and the grid fragments located outside the volume data; for the grid fragments located in the volume data, they are added to the original volume data as a new self-luminous medium to form new volume data, so that the self-luminous medium can participate in the solution of the rendering method; for the grid fragments located outside the volume data, after obtaining the rendering result for the newly formed volume data, the grid fragments located outside the volume data are mixed with the rendering result. In this way, the volume rendering algorithm can be improved to support the semi-transparent rendering algorithm, thereby realizing the combination of volume rendering and semi-transparent rendering to perform mixed rendering of volume data and surface data.

[0119] The technical solution provided by the embodiments of the present application is introduced below in conjunction with the accompanying drawings.

[0120] See also Figure 3 , is a schematic flow chart of a medical image rendering method provided in an embodiment of the present application, and the method may include the following steps:

[0121] Step S301: Acquire first volume data of a first object and surface mesh data of a second object.

[0122] The first object and the second object may be biological tissues or medical devices. In some embodiments, the first object is biological tissues, and the second object is a medical device acting on biological tissues. For example, the first object is a foot, that is, the first volume data of the foot is obtained; the second object is a medical device acting on the foot. Of course, the first object and the second object may also be biological tissues at the same part or the same position.

[0123] Normally, the volume data and surface mesh data are mixed and rendered in the same scene in order to see a more realistic tissue (obtained by rendering the volume data) through a semi-transparent instrument or tissue (obtained by rendering the surface mesh data). Therefore, the first object and the second object usually have a certain relationship. For example, the second object is a medical device that acts on the first object (biological tissue). In this case, after mixed rendering, the biological tissue can be seen through the semi-transparent medical device. For another example, the first object and the second object are both biological tissues, and the first object is a tissue located inside the second object. In this case, after mixed rendering, the internal biological tissue (i.e., the second object) can be seen through the semi-transparent tissue (i.e., the second object).

[0124] The first volume data is original volume data, which includes real media and virtual media.

[0125] Step S302: Perform semi-transparent rendering on the surface mesh data to obtain a first rendering result, where the first rendering result includes information of each mesh fragment.

[0126] In the process of mixed rendering of volume data and surface mesh data, a semi-transparent mesh rendering algorithm is first used to perform semi-transparent rendering on the surface mesh data to obtain a first rendering result; then the first rendering result is input into the volume rendering algorithm and rendered together with the volume data.

[0127] Among them, in the process of semi-transparent rendering of surface mesh data, the surface mesh data is first rasterized to project each geometric primitive (such as a triangle primitive) in the surface mesh data onto the screen, and the mesh fragments are obtained by judging which pixels fall within the projection graphics of the geometric primitives. Then, through the shading stage, the pixel point information, color information, depth information, etc. of each mesh fragment are obtained.

[0128] Color information may include information of four channels: R (Red), G (Green), B (Blue), and A (Alpha). The A channel refers to the opacity information of the grid element.

[0129] The depth information represents the depth of the mesh fragment in the Normalized Device Coordinate (NDC) coordinate system.

[0130] The pixel information may indicate the pixel coordinates where the grid fragment is located, that is, indicates at which pixel the grid fragment is located. Multiple grid fragments may be drawn at the same pixel on the screen.

[0131] After obtaining the color information, depth information and pixel information of each mesh fragment, coordinate conversion is performed according to the pixel information and depth information, and the mesh fragment is converted to the volume data coordinate system to determine whether the mesh fragment is inside the volume data or outside the volume data. In addition, depth sorting is performed for the mesh fragments under the same pixel according to the depth information of the mesh fragments to obtain depth sorting information to obtain the above-mentioned first rendering result.

[0132] The first rendering result includes information of each mesh fragment and depth sorting information. The information of the mesh fragment may include but is not limited to: color information, pixel information, depth information, and flag information for indicating whether it is inside the volume data.

[0133] According to the flag information indicating whether the mesh fragment is inside the volume data, it can be determined whether the mesh fragment is inside the volume data. In the embodiment of the present application, the mesh fragment inside the volume data is recorded as the first mesh fragment, and the mesh fragment outside the volume data is recorded as the second mesh fragment.

[0134] Step S303: Perform volume rendering on the second volume data according to the first rendering result to obtain a second rendering result, wherein the second volume data is volume data composed of the first volume data and the first mesh fragment, and the first mesh fragment is a mesh fragment located inside the first volume data.

[0135] It should be noted that the second volume data is new volume data generated by adding the first grid fragment as a new medium into the first volume data (ie, the original volume data).

[0136] The first volume data includes real media and virtual media. In the embodiment of the present application, the first grid element is defined as a grid medium, so the second volume data includes four types of media: grid media, real media, virtual media, and mixed media. Mixed media is a medium that is a mixture of real media and grid media.

[0137] The grid medium is a self-luminous medium. The first volume data does not include the self-luminous medium. The self-luminous medium is inserted into the volume data that originally does not include the self-luminous medium to form new volume data.

[0138] The grid medium does not affect the propagation path of light in the volume data.

[0139] For example, see Figure 4 The schematic diagram of the propagation path of light through new volume data provided by the embodiment of the present application is shown. After the light emitted from the virtual camera passes through the pixel points on the screen, it propagates in the volume data. The volume data includes the medium (i.e. Figure 4 ). Among them, Figure 4In the light propagation path shown, the light passes through medium 1, medium 2, medium 3, medium 4, medium 5 and medium 6 in sequence. Medium 1 and medium 4 are both real media, and the light undergoes surface scattering and volume scattering in medium 1 and medium 4; medium 3 and medium 6 are mixed media, and the light undergoes volume scattering and surface scattering in medium 3 and medium 6 respectively; medium 2 and medium 5 are virtual media and grid media respectively, and the light does not change the propagation path in medium 2 and medium 5.

[0140] It should be noted that for the sake of convenience, Figure 4 The shapes of the small circles in the new volume data are the same. However, in practical applications, small circles of different shapes can be used to represent different media types. For example, Figure 4 As shown, a small circle with a shadow represents a real medium, a small circle with a blank space represents a virtual medium, a small circle with a grid represents a grid medium, and a small circle with a grid and a shadow represents a mixed medium.

[0141] In a specific application, the second volume data can be volume rendered according to ray tracing. After constructing the ray, it can be determined whether the screen pixel point through which the ray passes has a grid fragment drawn according to the information of each grid fragment in the first rendering result (such as the pixel point information above), and if so, the ray is marked, and if not, the ray is not marked.

[0142] For marked light, the influence of self-luminous items needs to be considered, and it is necessary to determine whether each distance sampling passes through the grid medium. For unmarked light, the influence of self-luminous items does not need to be considered, and it is not necessary to determine whether it passes through the grid medium at each distance sampling.

[0143] After the light passes through the screen pixel, it propagates in the second volume data. The sampling point of each distance sampling is obtained by iterative distance sampling of the light.

[0144] Among them, for each distance sampling, if the light is a marked light, then according to the first rendering result, it is determined whether it passes through the grid medium during the distance sampling process; if it passes through the grid medium, the grid medium passed through is mixed to obtain a grid mixing result, and the color contribution value of the sampling point is determined according to the grid mixing result and the medium type of the sampling point; if it does not pass through the grid medium, the color contribution value of the sampling point is determined according to the medium type of the sampling point. If the light is an unmarked light, the color contribution value of the sampling point is determined according to the medium type of the sampling point.

[0145] According to the color contribution value of each sampling point, the color information of the screen pixel through which the light passes can be determined, thereby obtaining the second rendering result.

[0146] The second rendering result may include, but is not limited to: color information and depth information of volume data points (also called volume data pixels) on each screen pixel point, etc. The color information includes color and opacity, etc.

[0147] In some embodiments, a color mixing operation may be performed on all grid media through which the light passes according to the color information and depth information of the grid media in the first rendering result. The color mixing operation of the grid media may include color mixing and opacity mixing, and specifically, the following formula 3 and formula 4 may be used to perform color mixing and opacity mixing, respectively.

[0148] a i+1 =a i +(1-a i )*b i+1 (3)

[0149] C i+1 =C i +S i+1 *(1-a i )*b i+1 (4)

[0150] Among them, a i is the opacity of the final output of the i-th depth level, a i+1 is the opacity of the final output of the i+1th depth level; b i+1 represents the opacity of the mesh fragment at the i+1th depth level, C i+1 Indicates the final output color of the i+1th depth level, C i represents the final output color of the i-th depth level, S i+1 Indicates the color of the mesh fragment at the i+1th depth level. The initial values ​​of a and C are 0.

[0151] According to the depth information of each first grid fragment, the depth sorting result of all the first grid fragments (i.e., the grid medium) passed by the light can be determined; according to the depth sorting result, based on the color information (including color and opacity) of the grid fragment, the above formulas 3 and 4 are used to perform color mixing and opacity mixing in the direction of increasing depth.

[0152] It can be understood that the grid medium is the first grid fragment located inside the first volume data, so the color mixing and opacity mixing of the grid medium can be performed according to the depth information and color information of the first grid fragment in the first rendering result.

[0153] For example, see Figure 5The schematic diagram of the depth sorting of the grid medium provided by the embodiment of the present application is shown, assuming that the first grid fragment (i.e., the grid medium) through which the light passes includes frag1, frag2, frag3, and frag4. The depth sorting of these four first grid fragments is: flag1<flag2<flag3<flag4. Flag1 is located at the first depth level, flag2 is located at the first depth level, flag3 is located at the first depth level, and flag4 is located at the first depth level.

[0154] According to the color and opacity of these four first mesh fragments, they are mixed forward in sequence along the line of sight. Specifically, for the first depth level, i=0 at this time, the initial values ​​of a0 and C0 are 0, and the opacity of the final output of the first depth level is: a1=a0+(1-a0)*b1, b1 is the opacity of the mesh fragment (i.e., frag1) located in the first depth level; the color of the final output of the first depth level is: C1=C0+S1*(1-a0)*b1, S1 represents the color of the mesh fragment (i.e., frag1) located in the first depth level. That is, for the first depth level, according to the color and opacity of frag1, the above formulas 3 and 4 are used to obtain the opacity and color of the final output of the first depth level.

[0155] For the second depth level, at this time, i=1. The mesh fragment at the second depth level is frag2. At this time, the opacity of the final output of the second depth level is: a2=a1+(1-a1)*b2, b2 is the opacity of the mesh fragment at the second depth level (i.e., frag2); the color of the final output of the second depth level is: C2=C1+S2*(1-a1)*b2, S2 represents the color of the mesh fragment at the second depth level (i.e., frag2).

[0156] That is, for the second depth level, according to the color and opacity of frag2 and the color and opacity finally outputted by the previous depth level (i.e., the first depth level), the above formulas 3 and 4 are used to obtain the opacity and color finally outputted by the second depth level.

[0157] Similarly, for the third depth level, i=2, the mesh fragment at the third depth level is frag3. The opacity of the final output of the third depth level is: a3=a2+(1-a2)*b3, b3 is the opacity of the mesh fragment at the third depth level (i.e., frag3); the color of the final output of the third depth level is: C3=C2+S3*(1-a2)*b3, S3 represents the color of the mesh fragment at the third depth level (i.e., frag3).

[0158] For the 4th depth level, at this time, i=3. The mesh fragment at the 4th depth level is frag4. At this time, the opacity of the final output of the 4th depth level is: a4=a3+(1-a3)*b4, b4 is the opacity of the mesh fragment at the 4th depth level (i.e., frag4); the color of the final output of the 4th depth level is: C4=C3+S4*(1-a3)*b4, S4 represents the color of the mesh fragment at the 4th depth level (i.e., frag4).

[0159] exist Figure 5 In the example, frag1 to frag4 are mixed, and the obtained grid mixing result includes the final output color (ie, a4) and opacity (ie, C4) of the fourth depth level.

[0160] Step S304: According to the first rendering result, a color mixing operation is performed on the second grid fragment and the second rendering result to obtain a medical image, wherein the second grid fragment is a grid fragment located outside the first volume data.

[0161] The second rendering result is the result of rendering the second volume data by ray tracing, which may include color information and depth information of each volume data point (also known as a volume data pixel point). Exemplarily, the result of ray tracing the second volume data may include a depth texture and a color texture. The depth texture and the color texture are consistent with the size of the rendering window. The depth texture is used to store the depth information of the volume data point, and the color texture is used to store the color information of the volume data point. The color information of the volume data point may include the color of the volume data point and the opacity of the volume data point.

[0162] The first rendering result may include information of each mesh fragment and depth sorting information. As described above, the information of the mesh fragment may include color information, pixel information, depth information, and flag information indicating whether it is inside the volume data. For each mesh fragment, it can be determined whether the mesh fragment is the first mesh fragment or the second mesh fragment according to the flag information indicating whether it is inside the volume data.

[0163] According to the pixel point information of each second grid fragment, the second grid fragment located under the same screen pixel point can be determined. In addition, according to the depth sorting information, the depth sorting result of each second grid fragment located under the same screen pixel point can be obtained.

[0164] The output of ray tracing is usually a high dynamic range (HDR), so tone mapping is needed to convert HDR into a low dynamic range (LDR). In the tone mapping stage, the second mesh fragment can be mixed with the second rendering result according to the second rendering result (including depth texture and color texture, etc.) and the first rendering result.

[0165] In some embodiments, see Figure 6 The schematic diagram of the color mixing process of the second grid fragment and the rendering result provided in the embodiment of the present application is shown. The process of performing a color mixing operation on the second grid fragment and the second rendering result according to the first rendering result to obtain a medical image may include the following steps:

[0166] Step S601: perform depth sorting on the volume data points and the second grid fragments located under the same screen pixel.

[0167] In a specific application, the volume data and the second grid fragment located under the same screen pixel are determined based on the screen pixel where the volume data point is located and the pixel information of the second grid fragment; for the volume data point and the second grid fragment under the same screen pixel, if multiple second grid fragments have been depth-sorted, the volume data point can be inserted into the multiple second grid fragments that have been depth-sorted according to the depth of the volume data point; if the multiple second grid fragments have not been depth-sorted, the volume data point and the multiple grid fragments can be depth-sorted according to the depth of the volume data point and the depth of each grid fragment.

[0168] For example, see Figure 7 The schematic diagram of depth sorting of the second grid fragments and volume data points provided by the embodiment of the present application is shown, assuming that a certain screen pixel point includes frag1 to frag4 and PBVR pixel points. frag1 to frag4 are second grid fragments, and PBVR pixel points are volume data points obtained by rendering the second volume data using the PBVR algorithm.

[0169] exist Figure 7 In the example, assuming that frag1 to frag4 have been depth sorted, the depth sorting results of the four second grid fragments are: frag1 < frag2 < frag3 < frag4. According to the depth of the PBVR pixel, the PBVR pixel is inserted into frag1 to frag4. At this time, there are three situations for the relative position between the PBVR pixel and frag1 to frag4. When the depth of the PBVR pixel is less than that of frag1, the PBVR pixel is inserted in front of frag1, such as Figure 7As shown in case 1 in ; when the depth of the PBVR pixel is greater than the depth of some of the second grid fragments, and less than the depth of some of the second grid fragments, the PBVR pixel is inserted between frag1 and frag4. For example, assuming that the depth of the PBVR pixel is greater than the depth of frag1, but less than the depth of frag2, the PBVR pixel is inserted between frag1 and frag2, as shown in Figure 7 As shown in case 2 in ; when the depth of the PBVR pixel is greater than frag4, the PBVR pixel is inserted after frag4, such as Figure 7 As shown in case 3.

[0170] exist Figure 7 In the direction of increasing depth (i.e., the direction of sight), case 1 indicates that all the second grid fragments are on the back of the volume data point; case 2 indicates that some of the second grid fragments are in front of the volume data point, and some of the second grid fragments are on the back of the volume data point; case 3 indicates that all the second grid fragments are in front of the volume data point.

[0171] After the PBVR pixels and frag1 to frag4 are sorted in depth, multiple depth levels are formed. There are PBVR pixels or second grid fragments in each depth level. Figure 7 As shown, there are 5 depth levels in total. For case 1, the first depth level contains PBVR pixels.

[0172] Step S602: for each depth level after depth sorting, obtain the mixed opacity of the current depth level according to the opacity of the target object in the current depth level and the mixed opacity of the previous depth level; if the target object in the current depth level is a volume data point, obtain the mixed color of the current depth level according to the color of the target object in the current depth level and the mixed transparency of the previous depth level; if the target object in the current depth level is a second grid fragment, obtain the mixed color of the current depth level according to the color and opacity of the target object in the current depth level, and the mixed color and mixed transparency of the previous depth level; the depth of the current depth level is greater than the depth of the previous depth level.

[0173] It should be noted that the target object in the current depth level may be a second mesh fragment or a volume data point. Figure 7 In case 1, the target object in the first depth level is the PBVR pixel, and the target object in the third depth level is frag2.

[0174] After the volume data points and the second grid fragments are sorted by depth, the colors can be mixed forward in sequence along the line of sight (i.e., the direction of increasing depth). Different target objects in the current depth level use different color mixing formulas. If the target object in the current level is the second grid fragment, the above formulas 3 and 4 can be used to perform color mixing and opacity mixing, respectively; if the target object in the current level is a volume data point, the following formulas 5 and 6 can be used to perform color mixing and opacity mixing, respectively.

[0175] a i+1 =a i +(1-a i )*b i+1 (5)

[0176] C i+1 =P i+1 (1-a i ) (6)

[0177] Among them, b i+1 refers to the opacity of the volume data point in the i+1th depth level; P i+1 Refers to the color of the volume data point in the i+1th depth level.

[0178] It should be noted that Formula 3 is the same as Formula 5. When applied to the calculation of the second grid element, b i+1 refers to the opacity of the grid fragment in the i+1th depth level; when applied to volume data point calculations, b i+1 Refers to the opacity of the volume data point in the i+1th depth level.

[0179] For example, see Figure 7 In case 2, for the first depth level, i=0, and the initial values ​​of a0 and C0 are 0. The opacity of the final output of the first depth level is: a1=a0+(1-a0)*b1, where b1 is the opacity of the mesh fragment (i.e., frag1) located in the first depth level. If the target object in the first depth level is the second mesh fragment, the above formula 4 is used to calculate the mixed color. At this time, the color of the final output of the first depth level is: C1=C0+S1*(1-a0)*b1, where S1 represents the color of the mesh fragment (i.e., frag1) located in the first depth level. That is, for the first depth level, according to the color and opacity of frag1, the above formulas 3 and 4 are used to obtain the opacity (i.e., mixed opacity) and color (i.e., mixed color) of the final output of the first depth level.

[0180] For the second depth level, i=1. At this time, the target object of the second depth level is the volume data point, so the above formula 5 and formula 6 are used to calculate the mixed opacity and mixed color. Specifically, the opacity of the final output of the second depth level is: a2=a1+(1-a1)*b2, b2 is the opacity of the volume data point (i.e., PBVR pixel point) located in the second depth level, and a1 is the mixed opacity output of the previous depth level (i.e., the first depth level).

[0181] The final output color of the second depth level is: C2=P2(1-a1), where P2 represents the color of the volume data point (ie, PBVR pixel point) located in the second depth level.

[0182] For the third depth level, i=2. The target object at the third depth level is frag2. The opacity of the final output of the third depth level is: a3=a2+(1-a2)*b3, b3 is the opacity of the mesh fragment (i.e., frag2) at the third depth level; the color of the final output of the third depth level is: C3=C2+S3*(1-a2)*b3, S3 represents the color of the mesh fragment (i.e., frag2) at the third depth level.

[0183] Similarly, for the 4th depth level, i=3, the target object at the 4th depth level is frag3. The opacity of the final output of the 3rd depth level is: a4=a3+(1-a3)*b4, b4 is the opacity of the mesh fragment (i.e., frag3) at the 4th depth level; the color of the final output of the 4th depth level is: C4=C3+S4*(1-a3)*b4, S4 represents the color of the mesh fragment (i.e., frag3) at the 4th depth level.

[0184] For the fifth depth level, i=4, and the mesh fragment at the fifth depth level is frag4. At this time, the opacity of the fifth depth level is: a5=a4+(1-a4)*b5, b5 is the opacity of the mesh fragment at the fifth depth level (i.e., frag4); the color of the fifth depth level is: C5=C4+S5*(1-a4)*b5, S5 represents the color of the mesh fragment at the fifth depth level (i.e., frag4).

[0185] Step S603: output a medical image according to the mixed color and mixed opacity of the depth level with the greatest depth.

[0186] For example, in Figure 7In case 2, the depth level with the largest depth is the 5th depth level. At this time, the output medical image can be obtained according to the mixed color (ie, C5) and the mixed opacity (ie, a5) of the 5th depth level.

[0187] As can be seen from the above, the embodiment of the present application presents a mixed rendering of surface mesh data and volume data based on two methods, namely, volume rendering and semi-transparent rendering, so that semi-transparent biological tissue or medical device can be displayed in the biological tissue obtained by volume rendering. That is, in the rendered medical image, the biological tissue or medical device corresponding to the surface mesh data has a semi-transparent display effect, and through the semi-transparent biological tissue or medical device, the biological tissue obtained by volume rendering of the volume data can be observed.

[0188] Based on the above embodiments, in some embodiments, the PPLL algorithm may be used to perform semi-transparent rendering on the surface mesh data to obtain a first rendering result. At this time, the first rendering result may include a linked list and a texture image.

[0189] For example, see Figure 8 The schematic block diagram of the semi-transparent rendering process of the surface mesh data provided by the embodiment of the present application is shown. The process of performing semi-transparent rendering on the surface mesh data to obtain the first rendering result may include the following steps:

[0190] Step S801, perform semi-transparent rendering on the surface mesh data to obtain a first linked list and a first texture image, wherein the first linked list includes color information and depth information of each mesh fragment, a flag indicating whether the mesh fragment is inside the first volume data, and index information indicating the position of the mesh fragment last drawn on the screen pixel in the linked list, and the first texture image is used to indicate the position of the mesh fragment last drawn on the screen pixel in the linked list.

[0191] Among them, the PPLL algorithm can be used for semi-transparent rendering of surface mesh data. The PPLL algorithm records the depth information and color information of each mesh fragment through a linked list (List buffer) (the color is generally calculated through the Phong shading model); at the same time, according to the rendering order, for the mesh fragments drawn on the same pixel, a link is established in the List Buffer through the Index of each mesh fragment, that is, as long as the last mesh fragment drawn on a certain pixel is known, other mesh fragments on this pixel can be found; the mesh fragments on the same pixel are sorted by depth and then mixed to achieve a semi-transparent effect.

[0192] For example, see Fig. 9The chain table schematic diagram provided by the embodiment of the present application is shown, and the chain list (ListBuffer) stores information of multiple grid fragments such as frag0, frag1, frag2 and frag3. Among them, each grid fragment includes Index, Color (color information), Depth (depth information) and Flag (flag bit). The color information includes the four-channel color information (i.e., RGBA) of the grid fragment, and the depth information is the depth information in the NDC coordinate system. Flag is a flag bit used to indicate whether the grid fragment is inside the first volume data. Index is an index information used to indicate the position of the grid fragment drawn last time on the screen pixel point in the chain list.

[0193] The first texture image is a texture of the same size as the rendering window (the Head Image can be located), which can be used to store the position of the last grid fragment drawn on the screen pixel in the linked list.

[0194] For example, see Fig.10 The schematic diagram of the head image provided by the embodiment of the present application is shown, and four grid fragments are drawn on a certain pixel point in the head image (HeadImage), namely frag9, frag7, frag6 and frag1. The grid fragment drawing order of the pixel point is: frag1, frag6, frag7, frag9, that is, the last grid fragment drawn on the pixel point is frag9, and the first grid fragment drawn is frag1.

[0195] According to the header image, frag9 can be found in the List Buffer; according to the Index in frag9, frag7 can be found in the List Buffer; according to the Index in frag7, frag6 can be found in the List Buffer; according to the Index in frag6, frag1 can be found in the List Buffer.

[0196] Step S802: for each screen pixel, convert each grid fragment on the screen pixel into the coordinate system of the first volume data to determine whether the grid fragment is located inside the first volume data; if the grid fragment is located inside the first volume data, set the flag of the grid fragment to the first state; if the grid fragment is located outside the second volume data, set the flag of the grid fragment to the second state.

[0197] The first state can be true, and the second state can be false. That is, when the mesh fragment is located inside the first volume data, the Flag of the mesh fragment in the first linked list is set to true; when the mesh fragment is located outside the first volume data, the Flag of the mesh fragment in the first linked list is set to false. Later, according to the specific value of the Flag of the mesh fragment, it can be known whether the mesh fragment is the first mesh fragment or the second mesh fragment.

[0198] Step S803 : for each screen pixel, perform depth sorting on each grid fragment on the screen pixel according to the depth information of the grid fragment to obtain a depth sorting result.

[0199] Step S804: modify the index information of the first texture image and the mesh fragments in the first linked list according to the depth sorting result to obtain the second texture image and the second linked list.

[0200] The first texture image stores the position of the last drawn mesh fragment on a certain pixel in the linked list. The second texture image stores the position of the mesh fragment with the maximum or minimum depth on a certain pixel in the linked list, that is, according to the depth sorting result, the mesh fragment with the maximum or minimum depth on a certain pixel can be obtained, and the position of the mesh fragment with the maximum or minimum depth in the linked list is stored in the second texture image. In addition, the index information (i.e., Index) of the mesh fragment in the first linked list can be modified according to the depth sorting result to obtain the second linked list.

[0201] For example, see Fig.11 The schematic diagram of the new head image provided by the embodiment of the present application is shown, and the new head image (New Head Image) is sorted according to the depth result. Fig.10 The head image is modified.

[0202] Specifically, if Fig.10 As shown in the figure, frag9, frag7, frag6 and frag1 are drawn on a certain pixel. The depth sorting of these four grid fragments is performed, and the depth sorting result is: frag6>frag9>frag7>frag1, that is, frag6 has the largest depth and frag1 has the smallest depth. In addition, it can be determined through judgment that frag1 and frag6 are located outside the first volume data, and frag7 and frag9 are located inside the first volume data, that is, the Flag of frag1 and frag6 is set to false, and the Flag of frag7 and frag9 is set to true.

[0203] Assuming that the new head image stores the position of the grid fragment with the smallest depth on the pixel point in the linked list, then Fig.11The new header image stores the position of frag1 in the linked list. Fig.10 The header image stores the position of frag9 in the linked list.

[0204] In order to find the position of the last drawn mesh fragment in the linked list through the index in the linked list, after modifying the header image to obtain a new header image, the index of frag7, frag9 and frag6 in the linked list can be modified. Fig.10 In the example, the Index of frag1 is null, that is, frag1 is the first grid fragment drawn at the pixel point; Fig.11 In the example, the Index of frag1 is the position of frag7 in the List Buffer. According to the Index of frag1, frag7 can be found in the List Buffer. That is, according to the depth sorting result, the Index of frag1 is changed from null to the position of frag7.

[0205] Similarly, in Fig.11 In the example, the index of frag7 is changed to the position of frag9 in the List Buffer, the index of frag9 is changed to the position of frag6 in the List Buffer, and the index of frag6 is changed to a null value. At this time, after modifying the texture image and linked list according to the depth sorting result, Fig.11 The drawing order in is: frag6, frag9, frag7, frag1, that is, the last grid fragment drawn at this pixel is frag1, and the first grid fragment drawn is frag6.

[0206] The new head image is the second texture image. According to the depth sorting result, the index of the grid fragment on the same pixel in the first linked list is modified to obtain the second linked list. At this time, according to the second texture image, the last grid fragment drawn on a certain pixel is frag1. According to the index in frag1, frag7 can be found in the second linked list; according to the index of frag7 in the second linked list, frag9 can be found in the second linked list; according to the index of frag9 in the second linked list, frag6 can be found in the second linked list.

[0207] It should be noted that the first rendering result may include the second linked list and the second texture image, wherein the second linked list stores the color information and depth information of each mesh fragment, and according to the Index in the second linked list and the second texture image, the pixel information of the mesh fragment (i.e., at which pixel the mesh fragment is located) and the depth sorting information (or depth sorting result) of each mesh fragment on the same pixel can be obtained. The Flag of each mesh fragment in the second linked list is the flag information used to indicate whether it is inside the volume data.

[0208] As can be seen from the above, the embodiment of the present application performs semi-transparent rendering on the surface mesh data through the PPLL algorithm, so as to effectively utilize the linked list and texture image to store relevant information, thereby facilitating the subsequent mixed rendering of volume data and surface mesh data based on the relevant information.

[0209] Based on the above embodiments, in some embodiments, see Fig.12 The schematic block diagram of the volume rendering process based on ray tracing provided by the embodiment of the present application is shown. The process of performing volume rendering on the second volume data according to the first rendering result to obtain the second rendering result may include the following steps:

[0210] Step S1201: construct a ray and project the ray to the second volume data, the ray passing through the screen pixel. The second volume data includes a grid medium, a real medium and a mixed medium, the grid medium is the first grid element, the real medium is the first volume data, and the mixed medium is a mixture of the grid medium and the real medium.

[0211] Exemplarily, the constructed ray expression is ray=d*t+o, where d is the direction of the ray, o is the starting point of the ray, and t is the transmission time of the ray.

[0212] The light is used to sample the second volume data, so the light is constructed in the volume data coordinate system. After passing through the screen pixel point, the constructed light propagates in the second volume data.

[0213] The real medium is the medium in the original volume data (first volume data).

[0214] Step S1202: If a grid fragment is drawn on the screen pixel, mark the light.

[0215] After the ray is constructed, you can determine whether to consider the influence of medium self-luminescence in subsequent calculations by whether there are grid fragments drawn on the screen pixels through which the ray passes, and then determine whether to mark the ray. If there are grid fragments drawn on the screen pixels through which the ray passes, the ray is marked, indicating that the influence of medium self-luminescence needs to be considered in subsequent calculations. If there are no grid fragments drawn on the screen pixels through which the ray passes, the ray is not marked, indicating that the influence of medium self-luminescence does not need to be considered in subsequent calculations.

[0216] In some embodiments, whether to mark the light ray can be determined by determining whether the target pixel in the second texture image stores the position information of the grid fragment. The target pixel is a pixel in the second texture image corresponding to the screen pixel through which the light ray passes. If the target pixel in the second texture image stores the position information of the grid fragment, the light ray is marked, otherwise the light ray is not marked.

[0217] The second texture image stores the position of the last drawn grid fragment of a certain pixel in the linked list, so the position information of the grid fragment refers to the position of the grid fragment in the linked list.

[0218] In a specific application, the screen pixel points through which the light passes can be used to sample the new head image. If the value returned by the sampling is not 0, the light needs to be marked; if the value returned by the sampling is 0, the light does not need to be marked. If the value returned by the sampling is not 0, it means that the location information of the grid fragment is stored on the corresponding pixel point of the new head image.

[0219] In addition, after constructing the ray, the ray may be intersected with the bounding box of the second volume data to obtain the time Tmin when the ray enters the bounding box and the time Tmax when the ray leaves the bounding box.

[0220] Step S1203: perform iterative distance sampling on the light.

[0221] The maximum number of distance sampling can be preset. If a distance sampling is successful, the distance sampling counter is incremented by one until the distance sampling counter is greater than or equal to the maximum number, and then a ray tracing is completed.

[0222] The iterative distance sampling method is to generate a sampling step (step) through a random variable to gradually increase the light transmission time t (initial value is Tmin), and then calculate the iteration point, that is, t=t+step.

[0223] Step S1204: for each distance sampling, if the light is marked, determine whether the light passes through the grid medium between the first time point and the second time point according to the first rendering result; if so, perform a color mixing operation on the grid medium through which the light passes to obtain a grid color mixing result; determine the color contribution value of the sampling point according to the medium type of the sampling point obtained by the distance sampling and the grid color mixing result, and perform directional sampling according to the medium type of the sampling point. The first time point is the initial time of the distance sampling, and the second time point is determined according to the first time point and the sampling step.

[0224] If the light is marked, the effect of the medium's self-luminescence needs to be considered. The grid medium in the second volume data is a self-luminous medium, so it is necessary to consider whether the light passes through the grid medium.

[0225] In some embodiments, see Fig.13 The schematic flowchart of the process of finding the intersection of light and a grid medium provided by an embodiment of the present application is shown. The process of determining whether the light passes through the grid medium between the first time point and the second time point according to the first rendering result may include the following steps:

[0226] Step S1301: Determine a first position reached by a light ray at a first time point, and a second position reached by the light ray at a second time point.

[0227] The first time point can be t, and the second time point can be t+step, where step is the sampling step. Based on the light expression and the first time point t, the first position reached by the light at the first time point t can be determined; based on the light expression and the second time point t+step, the second position reached by the light at the second time point t+step can be determined.

[0228] Step S1302: Determine a first depth at a first position and a second depth at a second position.

[0229] The first position may be a certain volume data coordinate p1 in the second volume data, and the second position may be a certain volume data coordinate p2 in the second volume data. The depth z1 of the volume data coordinate p1 in the NDC coordinate system (i.e., the first depth of the first position) is determined, and the depth z2 of the volume data coordinate p2 in the NDC coordinate system (i.e., the second depth of the second position) is determined.

[0230] Step S1303: Project the first position and the second position onto the screen to obtain a first screen point of the first position on the screen and a second screen point of the second position on the screen.

[0231] Specifically, the first position and the second position are converted into a screen coordinate system, thereby obtaining a first screen point and a second screen point. The first screen point and the second screen point are screen coordinate points.

[0232] For example, see Fig.14A A schematic diagram of screen coordinate points provided in an embodiment of the present application is shown, in which the volume data coordinate point p1 (first position) and the volume data coordinate point p2 (i.e., the second position) are projected onto the screen to obtain screen coordinate points S1 and S2, where S1 is the first screen point corresponding to the first position, and S2 is the second screen point corresponding to the second position.

[0233] It should be noted that S1 and S2 may or may not overlap. Fig.14A is an example where S1 and S2 do not overlap.

[0234] Step S1304: when the first screen point and the second screen point coincide, a first target grid fragment is acquired according to information of each grid fragment; if the first target grid fragment is acquired, it is determined that the light has passed through the grid medium; if the first target grid fragment is not acquired, it is determined that the light has not passed through the grid medium; the first target grid fragment is a grid fragment located on the first screen point and having a depth between the first depth and the second depth.

[0235] It should be noted that the first screen point and the second screen point coincide with each other means that the distance between the first screen point and the second screen point is less than or equal to a preset threshold, and it is not necessarily required that the first screen point and the second screen point completely coincide with each other. In other words, if the first screen point and the second screen point almost coincide with each other (that is, the distance between the two screen points is less than or equal to the preset threshold), it can be considered that the first screen point and the second screen point coincide with each other.

[0236] If the first screen point and the second screen point coincide with each other, the grid fragment located on the first screen point can be determined based on the pixel point information of each grid fragment; then, based on the depth information of the grid fragment, a grid fragment whose depth is between the first depth and the second depth is found from the multiple grid fragments located on the first screen point as the first target grid fragment.

[0237] The first target grid fragment is the grid medium that the light passes through. That is, if the first target grid fragment can be found, it means that the light passes through the grid medium between the first time point and the second time point, and the grid medium that the light passes through is the found first target grid fragment; if the first target grid fragment cannot be found, it means that the light does not pass through the grid medium between the first time point and the second time point.

[0238] Step S1305: When the first screen point and the second screen point do not overlap, for each pixel sampling point on the line between the first screen point and the second screen point, obtain a candidate mesh fragment located on the pixel sampling point and with a depth between the first depth and the second depth according to information of each mesh fragment, and determine the distance between each candidate mesh fragment and the light.

[0239] Step S1306: If the second target grid fragment exists, determine that the light passes through the grid medium; if the second target grid fragment does not exist, determine that the light does not pass through the grid medium, and the second target grid fragment is a candidate grid fragment whose distance is less than or equal to the preset distance threshold.

[0240] If the first screen point and the second screen point do not overlap, connect the first screen point and the second screen point to form a line segment. The line segment is uniformly sampled, with the sampling starting point being the first screen point and the sampling end point being the second screen point. Each pixel sampling point on the screen represents a pixel coordinate. For example, Fig.14A As shown in FIG. 1 , the line connecting S1 and S2 is uniformly sampled to obtain pixel sampling points. The black pixels are the pixel sampling points.

[0241] For each pixel sampling point, multiple grid fragments located on the pixel point can be determined based on the pixel point information of each grid fragment; based on the depth information of the grid fragment, a grid fragment with a depth between a first depth and a second depth is found from the multiple grid fragments located on the pixel point as a candidate grid fragment.

[0242] For each candidate mesh fragment, according to the depth information and pixel information of the mesh fragment, the coordinates of the subsequent mesh fragments are converted to the volume data coordinates, and the distance from the candidate mesh fragment to the light ray is calculated; it is determined whether the distance between the candidate mesh fragment and the light ray is less than or equal to the preset distance threshold. If so, the candidate mesh fragment is used as the second target mesh fragment.

[0243] For example, assuming that the distance between the candidate mesh fragment and the light is D, if |D|≤e, then the candidate mesh fragment is determined to be the second target mesh fragment, that is, it is determined that the light intersects with the mesh fragment; otherwise, the candidate mesh fragment is not the second target mesh fragment, that is, it is determined that the light does not intersect with the mesh fragment. e is an error value, and the default value may be 0.01.

[0244] If the second target grid fragment can be found, it is considered that the light has passed through the grid medium between the first time point and the second time point; conversely, if the second target grid fragment cannot be found, it is considered that the light has not passed through the grid medium between the first time point and the second time point.

[0245] In other embodiments, the light rays may be intersected with the grid elements between the first time point and the second time point, mathematical calculations may be performed, and the intersection points may be recorded, so as to determine whether the light rays pass through the grid medium and the information of the grid medium passed through.

[0246] However, compared to the method of solving the intersection of light and grid elements through mathematical calculation, the above Fig.13In the manner shown, the intersection of the light ray and the grid fragment is determined by the first rendering result, which reduces the amount of calculation and increases the calculation speed.

[0247] If it is determined that the light passes through the grid medium between the first time point and the second time point, the information of the grid medium passed through is recorded, and according to the information of the grid medium passed through, the grid medium passed through by the light is mixed to obtain a grid color mixing result. The grid medium is a first grid fragment, and the information of the grid medium passed through may include depth information and color information of the first grid fragment. The color information includes color and opacity.

[0248] In some embodiments, in the process of performing a color mixing operation on the grid medium through which the light passes, the mixed opacity of the current depth level can be obtained for each target grid fragment after depth sorting according to the opacity of the target grid fragment in the current depth level and the mixed opacity of the previous depth level; the mixed color of the current depth level can be obtained according to the color and opacity of the target grid fragment in the current depth level, and the mixed color and mixed transparency of the previous depth level; the depth of the current depth level is greater than the depth of the previous depth level; the grid color mixing result includes the mixed color and mixed opacity of the depth level with the greatest depth; the grid medium through which the light passes includes at least one target grid fragment.

[0249] The target mesh fragment may be a first target mesh fragment or a second target mesh fragment.

[0250] Specifically, you can use the above formula 3 and formula 4 to perform color mixing and opacity mixing respectively, and then obtain the grid color mixing result. For details, please refer to the above Figure 5 The relevant contents of Formula 3 and Formula 4 will not be repeated here.

[0251] After obtaining the mesh color mixing result, the mesh color mixing result is recorded in MixMeshColor.

[0252] After obtaining the grid color mixing result, the second time point (ie, t+step) can be used for sampling to obtain the sampling point. According to the medium type of the sampling point and the grid color mixing result, the direct illumination information and indirect illumination information of the sampling point are calculated respectively, and then the color contribution value of the sampling point is determined.

[0253] The medium types of sampling points can include mixed medium, real medium and grid medium. Different medium types use different methods to calculate illumination information.

[0254] In some embodiments, if the sampling point is a real medium, direct illumination information of the sampling point is determined; and the color contribution value of the sampling point is determined according to the direct illumination information and the grid color mixing result.

[0255] That is to say, if the sampling point is a real medium, the grid color mixing result is used as the self-luminous information; by calculating the illumination contribution of the light source to the sampling point, the direct illumination information of the sampling point is obtained.

[0256] Each sampling point needs to mix the color contributed by direct illumination and the color contributed by self-illumination through albedo as the color contribution value of the sampling point. The color contribution value of the current sampling point is: (1-ρ(x i )*(L e )+ρ(x i )*L d ; L e is the self-luminous information, L d is the direct lighting information; ρ is the albedo.

[0257] If the sampling point is a grid medium, the self-luminous information of the sampling point is determined according to the grid color mixing result and the color information of the third target grid fragment corresponding to the sampling point; and the color contribution value of the sampling point is determined according to the self-luminous information.

[0258] Among them, the third target mesh fragment corresponding to the sampling point is obtained by sampling the first rendering result according to the depth and screen pixel coordinates of the sampling point. Specifically, according to the screen pixel coordinates of the sampling point and the pixel coordinates of each first mesh fragment in the first rendering result, all first mesh fragments located at the same screen pixel as the sampling point are determined; according to the depth of each first mesh fragment and the depth of the sampling point, a first mesh fragment whose depth is almost close to the depth of the sampling point is found among all first mesh fragments located at the same screen pixel as the sampling point as the third target mesh fragment. For example, the depth of the sampling point is z, and it is assumed that the depth of the first mesh fragment located at the same screen pixel as the sampling point is z3. If |z-z3|≤μ, the first mesh fragment is considered to be the third target mesh fragment. μ is the depth error, and the default value can be 0.001. The color of the third target mesh fragment (such as Color in the linked list) is recorded as SampleColor.

[0259] At this time, the self-luminous information L of the sampling point e =MixMeshColor+SampleColor.

[0260] It should be noted that the sampling point is a grid medium and does not need to calculate direct illumination information, that is, L d is 0. According to the formula (1-ρ(x i )*(L e )+ρ(x i )*L d , the color contribution value of the sampling point can be calculated.

[0261] If the sampling point is a mixed medium, the direct illumination information of the sampling point is determined, and the self-luminous information of the sampling point is determined according to the grid color mixing result and the color information of the third target grid fragment corresponding to the sampling point; the color contribution value of the sampling point is determined according to the self-luminous information and the direct illumination information.

[0262] That is, if the sampling point is a mixed medium, the illumination contribution of the light source to the sampling point is calculated to obtain direct illumination information; and, through L e =MixMeshColor+SampleColor, determine the self-luminous information of the sampling point; finally, according to the formula (1-ρ(x i )*(L e )+ρ(x i )*L d , the color contribution value of the sampling point can be calculated.

[0263] Step S1205: For each distance sampling, if the light does not pass through the grid medium between the first time point and the second time point, or the light is not marked, determine the color contribution value of the sampling point according to the medium type of the sampling point obtained by the distance sampling, and perform directional sampling according to the medium type of the sampling point.

[0264] In some embodiments, if the sampling point is a real medium, the direct illumination information of the sampling point can be determined; based on the direct illumination information, the color contribution value of the sampling point is determined. At this time, the self-luminous information of the sampling point is 0. According to the formula (1-ρ(x i )*(L e )+ρ(x i )*L d , the color contribution value of the sampling point can be calculated.

[0265] If the sampling point is a grid medium, the self-luminous information of the sampling point is determined according to the color information of the third target grid element corresponding to the sampling point; and the color contribution value of the sampling point is determined according to the self-luminous information. e =SampleColor, no need to calculate direct lighting information, that is, L d is 0; according to the formula (1-ρ(x i )*(L e )+ρ(x i )*L d , the color contribution value of the sampling point can be calculated.

[0266] If the sampling point is a mixed medium, determine the direct illumination information of the sampling point, determine the self-luminous information of the sampling point according to the color information of the third target grid element corresponding to the sampling point, and determine the color contribution value of the sampling point according to the self-luminous information and direct illumination information. e=SampleColor, L d Not 0; According to the formula (1-ρ(x i )*(L e )+ρ(x i )*L d , the color contribution value of the sampling point can be calculated.

[0267] After each successful distance sampling, direction sampling is required to determine the direction of the light after passing through the medium. The current ray tracing algorithm already has an algorithm for the direction of the light, and the embodiment of the present application only needs to determine the scattering mode. Different scattering modes correspond to different algorithms for the emission mode.

[0268] In some embodiments, during the process of directional sampling according to the medium type of the sampling point, if the sampling point is a real medium or a mixed medium, the scattering mode of the sampling point is determined according to the normal vector of the real medium, and the scattering mode of the sampling point is volume scattering or surface scattering. If the sampling point is a grid medium, the propagation direction of the light is maintained.

[0269] That is to say, if the sampling point is a real medium, the normal vector of the sampling point is used to determine whether the light is scattered by volume or surface at the sampling point; if the sampling point is a grid medium, the light path does not change after the light passes through the medium; if the sampling point is a mixed medium, the scattering method of the sampling point is determined according to the normal vector of the real medium part in the mixed medium. Through the above directional sampling rules, the light propagation path can be kept consistent in the newly added grid medium and the original volume data.

[0270] Step S1206: After the light distance sampling is completed, the color information of the screen pixel is obtained according to the color contribution value of each sampling point.

[0271] That is to say, by accumulating the color contribution value of each sampling point, the color information of the screen point through which the light passes can be obtained.

[0272] In the embodiment of the present application, the mesh fragment located inside the volume data is added to the volume data as a new medium, so as to add the self-luminous medium (i.e., the first mesh fragment) to the volume data that originally does not contain the self-luminous medium, and ray tracing is performed based on the added new medium and the original volume data; during the ray tracing process, the mesh medium through which the light passes is mixed to achieve a semi-transparent display effect of the pixel point. In this way, the result of semi-transparent rendering is effectively used as the self-luminous medium to participate in the solution of the rendering equation in the volume rendering process, thereby achieving mixed rendering of volume data and surface mesh data.

[0273] Based on the above embodiments, after each distance sampling, it is usually necessary to determine the medium type of the sampling point, that is, to determine whether the sampling point is a real medium, a mixed medium or a grid medium.

[0274] In some embodiments, the volume data may be preprocessed to mark the grid medium or the mixed medium in the volume data, so that in the subsequent range sampling process, after the volume data is sampled, the medium type of the sampling point may be determined according to the mark.

[0275] In other embodiments, the medium type of the volume data does not need to be marked in advance, but the first rendering result may be sampled to determine whether the sampling point is a grid medium or a mixed medium.

[0276] Specifically, after obtaining the sampling point and before determining the color contribution value of the sampling point, the third depth of the sampling point and the target screen pixel corresponding to the sampling point can also be determined; based on the information of each grid fragment, the third target grid fragment is obtained, and the third target grid fragment is located on the target screen pixel, and the difference between the depth and the third depth is less than or equal to the preset depth threshold. The grid fragment; if the opacity of the third target grid fragment is zero, it is determined that the sampling point is a grid medium; if the opacity of the third target grid fragment is not zero, it is determined that the sampling point is a mixed medium.

[0277] That is, according to the target screen pixel of the sampling point and the pixel coordinates of each first grid fragment in the first rendering result, all first grid fragments located at the same screen pixel as the sampling point are determined; according to the depth of each first grid fragment and the third depth of the sampling point, among all first grid fragments located at the same screen pixel as the sampling point, a first grid fragment whose depth is almost close to the third depth of the sampling point is found as the third target grid fragment. For example, the third depth of the sampling point is z, and it is assumed that the depth of the first grid fragment located at the same screen pixel as the sampling point is z3. If |z-z3|≤μ, the first grid fragment is considered to be the third target grid fragment. μ is the depth error (i.e., the preset depth threshold), and the default value may be 0.001.

[0278] It should be noted that in order to ensure that the grid medium does not affect the light path, the opacity of the mixed medium still uses the opacity of the original volume data corresponding to the sampling point.

[0279] If the third target grid fragment cannot be obtained, it is determined that the sampling point is neither a grid medium nor a mixed medium.

[0280] It is understandable that after each distance sampling to obtain a sampling point, it is possible to determine whether the sampling point is a grid medium or a mixed medium by using the above medium type determination method.

[0281] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0282] See also Fig. 14B The schematic diagram of the hybrid rendering effect provided by the embodiment of the present application is shown, which is a hybrid display effect of PBVR and translucent grid. It may include a foot and a medical device embedded in the foot with a translucent effect. In this way, by rendering the medical device with a translucent effect, the user can better view the foot.

[0283] Corresponding to the medical image rendering method described in the above embodiment, Fig.15 A schematic block diagram of the structure of a medical image rendering device provided in an embodiment of the present application is shown. For ease of explanation, only the parts related to the embodiment of the present application are shown.

[0284] Reference Fig.15 , the device comprises:

[0285] An acquisition module 1501 is used to acquire first volume data of a first object and surface mesh data of a second object;

[0286] A semi-transparent mesh rendering module 1502 is used to perform semi-transparent rendering on the surface mesh data to obtain a first rendering result, where the first rendering result includes information of each mesh fragment;

[0287] A volume rendering module 1503, configured to perform volume rendering on second volume data according to the first rendering result to obtain a second rendering result, wherein the second volume data is volume data formed by the first volume data and a first mesh fragment, and the first mesh fragment is a mesh fragment located inside the first volume data;

[0288] The color blending module 1504 is used to perform a color blending operation on the second mesh fragment and the second rendering result according to the first rendering result to obtain a medical image, where the second mesh fragment is a mesh fragment located outside the first volume data.

[0289] In some possible implementations, the volume rendering module 1503 is specifically used to: construct a ray and project the ray to the second volume data, the ray passing through the screen pixel; if a grid fragment is drawn on the screen pixel, mark the ray; perform iterative distance sampling on the ray; for each distance sampling, if the ray is marked, determine whether the ray passes through the grid medium between the first time point and the second time point according to the first rendering result; if so, perform a color mixing operation on the grid medium passed by the ray to obtain a grid color mixing result; determine the color contribution value of the sampling point according to the medium type of the sampling point obtained by distance sampling and the grid color mixing result, and perform directional sampling according to the medium type of the sampling point; for each distance sampling, if the ray does not pass through the grid medium between the first time point and the second time point, or the ray is not marked, determine the color contribution value of the sampling point according to the medium type of the sampling point obtained by distance sampling, and perform directional sampling according to the medium type of the sampling point; after the ray distance sampling is completed, obtain the color information of the screen pixel according to the color contribution value of each sampling point;

[0290] Among them, the second volume data includes grid medium, real medium and mixed medium, the grid medium is the first grid element, the real medium is the first volume data, the mixed medium is a mixture of the grid medium and the real medium, the first time point is the initial time of the distance sampling, and the second time point is determined based on the first time point and the sampling step.

[0291] In some possible implementations, the volume rendering module 1503 is specifically used to: determine a first position reached by a ray of light at a first point in time, and a second position reached by the ray of light at a second point in time; determine a first depth of the first position and a second depth of the second position; project the first position and the second position onto a screen to obtain a first screen point on the screen for the first position and a second screen point on the screen for the second position; when the first screen point and the second screen point coincide, acquire a first target grid element according to information of each grid element; if the first target grid element is acquired, determine that the ray of light has passed through the grid medium; if the first target grid element is not acquired, determine that the ray of light has not passed through the grid medium; A target grid fragment is a grid fragment located on a first screen point and having a depth between a first depth and a second depth; when the first screen point and the second screen point do not coincide, for each pixel sampling point on a line connecting the first screen point and the second screen point, a candidate grid fragment located on the pixel sampling point and having a depth between the first depth and the second depth is obtained according to information of each grid fragment, and a distance between each candidate grid fragment and the light is determined; if a second target grid fragment exists, it is determined that the light passes through a grid medium; if the second target grid fragment does not exist, it is determined that the light does not pass through the grid medium, and the second target grid fragment is a candidate grid fragment whose distance is less than or equal to a preset distance threshold.

[0292] In some possible implementations, the volume rendering module 1503 is specifically used to: for each target mesh fragment after depth sorting, obtain the mixed opacity of the current depth level according to the opacity of the target mesh fragment in the current depth level and the mixed opacity of the previous depth level; obtain the mixed color of the current depth level according to the color and opacity of the target mesh fragment in the current depth level, and the mixed color and mixed transparency of the previous depth level; the depth of the current depth level is greater than the depth of the previous depth level; the mesh color mixing result includes the mixed color and mixed opacity of the depth level with the greatest depth; the mesh medium through which the light passes includes at least one target mesh fragment.

[0293] In some possible implementations, the volume rendering module 1503 is specifically used to: if the sampling point is a real medium, determine the direct illumination information of the sampling point; determine the color contribution value of the sampling point based on the direct illumination information and the grid color mixing result; if the sampling point is a grid medium, determine the self-luminous information of the sampling point based on the grid color mixing result and the color information of the third target grid fragment corresponding to the sampling point; determine the color contribution value of the sampling point based on the self-luminous information; if the sampling point is a mixed medium, determine the direct illumination information of the sampling point, determine the self-luminous information of the sampling point based on the grid color mixing result and the color information of the third target grid fragment corresponding to the sampling point; determine the color contribution value of the sampling point based on the self-luminous information and the direct illumination information.

[0294] In some possible implementations, the volume rendering module 1503 is specifically used to: if the sampling point is a real medium, determine the direct illumination information of the sampling point; determine the color contribution value of the sampling point based on the direct illumination information; if the sampling point is a grid medium, determine the self-luminous information of the sampling point based on the color information of the third target grid fragment corresponding to the sampling point; determine the color contribution value of the sampling point based on the self-luminous information; if the sampling point is a mixed medium, determine the direct illumination information of the sampling point, determine the self-luminous information of the sampling point based on the color information of the third target grid fragment corresponding to the sampling point; determine the color contribution value of the sampling point based on the self-luminous information and the direct illumination information.

[0295] In some possible implementations, the volume rendering module 1503 is further used to: determine the third depth of the sampling point and the target screen pixel corresponding to the sampling point; obtain a third target grid fragment based on information of each grid fragment, where the third target grid fragment is located on the target screen pixel and the difference between the depth and the third depth is less than or equal to a preset depth threshold; if the opacity of the third target grid fragment is zero, determine that the sampling point is a grid medium; if the opacity of the third target grid fragment is not zero, determine that the sampling point is a mixed medium.

[0296] In some possible implementations, the volume rendering module 1503 is specifically used to: if the sampling point is a real medium or a mixed medium, determine the scattering mode of the sampling point according to the normal vector of the real medium, and the scattering mode of the sampling point is volume scattering or surface scattering; if the sampling point is a grid medium, maintain the propagation direction of the light.

[0297] In some possible implementations, the semi-transparent mesh rendering module 1502 is specifically used to: perform semi-transparent rendering on the surface mesh data to obtain a first linked list and a first texture image, the first linked list including color information and depth information of each mesh fragment, a flag indicating whether the mesh fragment is inside the first volume data, and index information indicating the position of the mesh fragment last drawn on the screen pixel in the linked list, and the first texture image is used to indicate the position of the mesh fragment last drawn on the screen pixel in the linked list; for each screen pixel, convert each mesh fragment on the screen pixel into the coordinate system of the first volume data to determine whether the mesh fragment is located inside the first volume data; if the mesh fragment is located inside the first volume data, set the flag of the mesh fragment to a first state; if the mesh fragment is located outside the second volume data, set the flag of the mesh fragment to a second state; for each screen pixel, perform depth sorting on each mesh fragment on the screen pixel according to the depth information of the mesh fragment to obtain a depth sorting result; according to the depth sorting result, modify the first texture image and the index information of the mesh fragment in the first linked list to obtain a second texture image and a second linked list.

[0298] In some possible implementations, the semi-transparent grid rendering module 1502 is specifically used to: if the target pixel point in the second texture image stores the position information of the grid element, then mark the light, and the target pixel point is the pixel point in the second texture image corresponding to the screen pixel point through which the light passes.

[0299] In some possible implementations, the second rendering result includes color information of the volume data point on the screen pixel point, and the color information of the volume data includes the color of the volume data point and the opacity of the volume data point; the color blending module 1504 is specifically used to: perform depth sorting on the volume data points and the second grid fragment located under the same screen pixel point; for each depth level after the depth sorting, obtain the mixed opacity of the current depth level according to the opacity of the target object in the current depth level and the mixed opacity of the previous depth level; if the target object in the current depth level is the volume data point, obtain the mixed color of the current depth level according to the color of the target object in the current depth level and the mixed transparency of the previous depth level; if the target object in the current depth level is the second grid fragment, obtain the mixed color of the current depth level according to the color and opacity of the target object in the current depth level, and the mixed color and mixed transparency of the previous depth level; the depth of the current depth level is greater than the depth of the previous depth level; and output a medical image according to the mixed color and mixed opacity of the depth level with the greatest depth.

[0300] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / modules are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0301] Fig.16 This is a schematic diagram of the structure of an electronic device provided by an embodiment of the present application. Fig.16 As shown, the electronic device 16 of this embodiment includes: at least one processor 160 ( Fig.16 Only one is shown), a memory 161, and a computer program 162 stored in the memory 161 and executable on the at least one processor 160, wherein the processor 160 implements the steps in any of the above-mentioned charging pile identification method embodiments when executing the computer program 162.

[0302] The electronic device may include, but is not limited to, a processor 160 and a memory 161. Those skilled in the art will appreciate that Fig.16 It is only an example of the electronic device 16 and does not constitute a limitation on the electronic device 16. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, it may also include input and output devices, network access devices, etc.

[0303] The processor 160 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0304] In some embodiments, the memory 161 may be an internal storage unit of the electronic device 16, such as a hard disk or memory of the electronic device 16. In other embodiments, the memory 161 may also be an external storage device of the electronic device 16, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device 16. Further, the memory 161 may also include both an internal storage unit of the electronic device 16 and an external storage device. The memory 161 is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as the program code of the computer program, etc. The memory 161 may also be used to temporarily store data that has been output or is to be output.

[0305] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0306] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium can at least include: any entity or device that can carry the computer program code to the camera / terminal device, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, RandomAccess Memory), electric carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0307] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0308] An embodiment of the present application provides a computer program product. When the computer program product runs on an electronic device, the electronic device can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0309] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0310] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0311] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0312] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0313] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0314] In the embodiments provided in the present application, it should be understood that the disclosed devices / network equipment and methods can be implemented in other ways. For example, the device / network equipment embodiments described above are merely schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0315] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0316] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A medical image rendering method, characterized in that: include: Acquire first volume data of a first object and surface mesh data of a second object; Performing semi-transparent rendering on the surface mesh data to obtain a first rendering result, wherein the first rendering result includes information of each mesh fragment; performing volume rendering on second volume data according to the first rendering result to obtain a second rendering result, wherein the second volume data is volume data formed by the first volume data and a first mesh fragment, and the first mesh fragment is the mesh fragment located inside the first volume data; According to the first rendering result, a color blending operation is performed on a second grid fragment and the second rendering result to obtain a medical image, wherein the second grid fragment is the grid fragment located outside the first volume data.

2. The method according to claim 1, characterized in that The performing volume rendering on the second volume data according to the first rendering result to obtain a second rendering result includes: Constructing a ray, and projecting the ray toward the second volume data, wherein the ray passes through screen pixels; If the grid fragment is drawn on the screen pixel, marking the light; Iteratively sampling the distance of the light; For each distance sampling, if the light is marked, determine whether the light passes through a grid medium between a first time point and a second time point according to the first rendering result; if so, perform a color mixing operation on the grid medium passed by the light to obtain a grid color mixing result; determine a color contribution value of the sampling point according to a medium type of the sampling point obtained by distance sampling and the grid color mixing result, and perform directional sampling according to the medium type of the sampling point; For each distance sampling, if the light does not pass through the grid medium between the first time point and the second time point, or the light is not marked, determine the color contribution value of the sampling point according to the medium type of the sampling point obtained by distance sampling, and perform direction sampling according to the medium type of the sampling point; After the light distance sampling is completed, the color information of the screen pixel point is obtained according to the color contribution value of each sampling point; Among them, the second volume data includes grid medium, real medium and mixed medium, the grid medium is the first grid fragment, the real medium is the first volume data, the mixed medium is a mixture of the grid medium and the real medium, the first time point is the initial time of distance sampling, the second time point is determined based on the first time point and the sampling step, and the second rendering result includes the color information of the screen pixel.

3. The method according to claim 2, characterized in that Determining, according to the first rendering result, whether the light passes through a grid medium between a first time point and a second time point includes: Determine a first position reached by the light at the first time point, and a second position reached by the light at the second time point; determining a first depth at the first location and a second depth at the second location; Projecting the first position and the second position onto a screen to obtain a first screen point of the first position on the screen and a second screen point of the second position on the screen; When the first screen point and the second screen point coincide with each other, acquiring a first target grid fragment according to information of each grid fragment; if the first target grid fragment is acquired, determining that the light has passed through the grid medium; if the first target grid fragment is not acquired, determining that the light has not passed through the grid medium; the first target grid fragment is a grid fragment located on the first screen point and at a depth between the first depth and the second depth; When the first screen point and the second screen point do not overlap, for each pixel sampling point on the line connecting the first screen point and the second screen point, according to information of each of the grid fragments, obtaining a candidate grid fragment located on the pixel sampling point and having a depth between the first depth and the second depth, and determining a distance between each of the candidate grid fragments and the light; If the second target grid fragment exists, it is determined that the light passes through the grid medium; if the second target grid fragment does not exist, it is determined that the light does not pass through the grid medium, and the second target grid fragment is a candidate grid fragment whose distance is less than or equal to the preset distance threshold.

4. The method according to claim 2, characterized in that: Performing a color mixing operation on the grid medium through which the light passes to obtain a grid color mixing result, including: For each target mesh fragment after depth sorting, obtain the mixed opacity of the current depth level according to the opacity of the target mesh fragment in the current depth level and the mixed opacity of the previous depth level; obtain the mixed color of the current depth level according to the color and opacity of the target mesh fragment in the current depth level and the mixed color and mixed transparency of the previous depth level; the depth of the current depth level is greater than the depth of the previous depth level; The grid color blending result includes the blended color and blended opacity of the depth level with the greatest depth; The grid medium through which the light passes includes at least one of the target grid fragments.

5. The method according to claim 2, characterized in that: Determining the color contribution value of the sampling point according to the medium type of the sampling point obtained by distance sampling and the grid color mixing result, including: If the sampling point is the real medium, determining direct illumination information of the sampling point; determining a color contribution value of the sampling point according to the direct illumination information and a grid color mixing result; If the sampling point is the grid medium, determining the self-luminous information of the sampling point according to the grid color mixing result and the color information of the third target grid fragment corresponding to the sampling point; and determining the color contribution value of the sampling point according to the self-luminous information; If the sampling point is the mixed medium, determine the direct illumination information of the sampling point, and determine the self-luminous information of the sampling point according to the grid color mixing result and the color information of the third target grid fragment corresponding to the sampling point; determine the color contribution value of the sampling point according to the self-luminous information and the direct illumination information.

6. The method according to claim 2, characterized in that Determining the color contribution value of the sampling point according to the medium type of the sampling point obtained by distance sampling includes: If the sampling point is the real medium, determining direct illumination information of the sampling point; and determining a color contribution value of the sampling point according to the direct illumination information; If the sampling point is the grid medium, determining the self-luminous information of the sampling point according to the color information of the third target grid fragment corresponding to the sampling point; and determining the color contribution value of the sampling point according to the self-luminous information; If the sampling point is the mixed medium, determine the direct illumination information of the sampling point, and determine the self-luminous information of the sampling point according to the color information of the third target grid fragment corresponding to the sampling point; determine the color contribution value of the sampling point according to the self-luminous information and the direct illumination information.

7. The method according to claim 1, characterized in that Performing semi-transparent rendering on the surface mesh data to obtain a first rendering result includes: Performing semi-transparent rendering on the surface mesh data to obtain a first linked list and a first texture image, wherein the first linked list includes color information and depth information of each mesh fragment, a flag bit indicating whether the mesh fragment is inside the first volume data, and index information indicating a position of a mesh fragment drawn last time on a screen pixel point in the linked list, and the first texture image is used to indicate a position of a mesh fragment drawn last time on the screen pixel point in the linked list; For each of the screen pixels, convert each of the grid fragments on the screen pixel into the coordinate system of the first volume data to determine whether the grid fragment is located inside the first volume data; if the grid fragment is located inside the first volume data, set the flag bit of the grid fragment to a first state; if the grid fragment is located outside the second volume data, set the flag bit of the grid fragment to a second state; For each of the screen pixels, according to the depth information of the grid fragments, depth sorting of each of the grid fragments on the screen pixel is performed to obtain a depth sorting result; According to the depth sorting result, the first texture image and the index information of the mesh fragments in the first linked list are modified to obtain a second texture image and a second linked list.

8. The method according to claim 1, characterized in that The second rendering result includes color information of the volume data point on the screen pixel point, and the color information of the volume data includes the color of the volume data point and the opacity of the volume data point; According to the first rendering result, a second grid fragment and the second rendering result are subjected to a color mixing operation to obtain a medical image, including: Depth-sorting the volume data points and the second grid fragments located under the same screen pixel point; For each depth level after depth sorting, the mixed opacity of the current depth level is obtained according to the opacity of the target object in the current depth level and the mixed opacity of the previous depth level; if the target object in the current depth level is the volume data point, the mixed color of the current depth level is obtained according to the color of the target object in the current depth level and the mixed transparency of the previous depth level; if the target object in the current depth level is the second grid fragment, the mixed color of the current depth level is obtained according to the color and opacity of the target object in the current depth level, and the mixed color and mixed transparency of the previous depth level; the depth of the current depth level is greater than the depth of the previous depth level; The medical image is output according to the mixed color and mixed opacity of the depth level with the greatest depth.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 8 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images