Hybrid rendering method and device, electronic equipment and storage medium

Abstract

The invention is suitable for the technical field of computer graphics, and provides a mixed rendering method and device for surface grid data and volume data, electronic equipment and a computer readable storage medium, and the method comprises the steps: carrying out the coordinate conversion and rasterization of the surface grid data, and obtaining a plurality of fragments, each fragment is associated with one pixel in the target image; generating a linked list of each pixel in the target image, wherein the linked list comprises information of a fragment associated with the corresponding pixel; removing information of a first fragment in each linked list, wherein the first fragment is located in the volume data; performing hybrid rendering on the plurality of individual data slices and the first fragment to obtain a hybrid rendering result of each pixel in the target image, the plurality of individual data slices being obtained by slicing the volume data; the mixture drawing result of each pixel in the target image is inserted into the corresponding linked list according to the depth; and for each pixel in the target image, performing color mixing on the fragment elements in the linked list and the mixture drawing result to obtain the target image.

Description

Technical Field

[0001] The present application belongs to the field of computer graphics, and in particular, relates to a hybrid rendering method and device of surface mesh data and volume data, an electronic device, and a computer-readable storage medium. Background Art

[0002] Three-dimensional visualization is the use of computer graphics to visualize three-dimensional volume data, which can be applied to many fields, such as medical imaging, maps, etc. Taking medical image visualization as an example, computer graphics is used to reconstruct 3D volume data obtained from medical imaging data obtained by technologies such as magnetic resonance imaging (MRI), computed tomography (CT), positron emission tomography (PET), and ultrasound (US) to perform 3D visualization. It is widely used in diagnosis, treatment, and surgery scenarios, and can display medical imaging data more intuitively and effectively so as to extract information from it.

[0003] The process of visualizing and rendering 3D volume data is called volume rendering, which can be divided into direct volume rendering and indirect volume rendering. Indirect volume rendering is to first extract isosurfaces from volume data, and then draw polygonal meshes composed of isosurfaces; direct volume rendering generates a 2D projection image of 3D volume data based on the color and opacity of the volume data.

[0004] The Per-Pixel Linked List (PPLL) method is an order-independent semi-transparent mesh rendering algorithm used for semi-transparent rendering of surface mesh data.

[0005] Direct volume rendering is used to render volume data, while PPLL is used to render surface mesh data, which belong to different algorithm fields. However, in practical applications, there are often scenarios where surface mesh data and volume data need to be rendered simultaneously. For example, in medical image visualization, medical devices are represented by semi-transparent meshes, and medical devices and human tissues need to be displayed simultaneously in the visualization. Direct volume rendering itself does not support the rendering of surface mesh data, and PPLL does not support the rendering of volume data. Therefore, the above-mentioned requirement of simultaneously rendering surface mesh data and volume data cannot be met. Summary of the invention

[0006] The embodiments of the present application provide a method and device for hybrid rendering of surface mesh data and volume data, an electronic device, and a computer-readable storage medium, which can solve the problem in the related art that surface mesh data and volume data cannot be drawn simultaneously.

[0007] In a first aspect, an embodiment of the present application provides a method for hybrid rendering of surface mesh data and volume data, including: performing coordinate transformation and rasterization on the surface mesh data to obtain multiple fragments, each fragment being associated with a pixel in a target image; generating a linked list of each pixel in the target image, the linked list including information of the fragments associated with the corresponding pixel; removing information of the first fragment in each linked list, the first fragment being located inside the volume data; performing hybrid body drawing on multiple volume data slices and the first fragment to obtain a hybrid body drawing result for each pixel in the target image, the multiple volume data slices being obtained by slicing the volume data; inserting the hybrid body drawing result for each pixel in the target image into a corresponding linked list according to depth; for each pixel in the target image, mixing the fragments in the linked list and the hybrid body drawing result to obtain the target image.

[0008] In a possible implementation manner of the first aspect, performing mixed volume rendering on multiple volume data slices and a first fragment to obtain a mixed volume rendering result for each pixel in a target image includes: sorting the volume data slices and the first fragment according to depth to determine a volume data slice adjacent to the first fragment in depth; rendering the volume data slices to obtain a slice rendering result of the volume data slices; blending the first fragment into a slice rendering result of an adjacent volume data slice; and blending the slice rendering results to obtain a mixed volume rendering result for each pixel in the target image.

[0009] In a possible implementation manner of the first aspect, blending the first fragment into a slice rendering result of an adjacent volume data slice includes: blending the first fragment into a slice rendering result of a volume data slice that is adjacent in depth and has an earlier iteration order in a process of blending the slice rendering results.

[0010] In a possible implementation manner of the first aspect, inserting the mixed body drawing result of each pixel in the target image into the corresponding linked list according to the depth includes: for each pixel in the target image, sorting the mixed body drawing result and the fragments in the corresponding linked list according to the depth to obtain a depth sorting result; and inserting the mixed body drawing result into the corresponding linked list according to the depth sorting result.

[0011] In a possible implementation of the first aspect, removing the information of the first fragment in each linked list includes: for each linked list, judging whether the first fragment exists according to the depth of each fragment therein; when the first fragment exists, removing the information of the first fragment from the linked list and adaptively updating the relevant index value.

[0012] In a possible implementation manner of the first aspect, before mixing the plurality of volume data slices with the first fragment for volume rendering, the method further includes: slicing the volume data using half-angle slice volume rendering to obtain the plurality of volume data slices.

[0013] In a possible implementation manner of the first aspect, the method further includes: constructing a mask image of the same size as the volume data; and modifying the value of the corresponding voxel of the voxel corresponding to the first fragment in the mask image to the number of the first fragment.

[0014] In a second aspect, an embodiment of the present application provides a hybrid rendering device of surface mesh data and volume data, including: a rendering module, used to perform coordinate transformation and rasterization on the surface mesh data to obtain multiple fragments, each fragment being associated with a pixel in the target image; a generation module, used to generate a linked list of each pixel in the target image, the linked list including information of the fragments associated with the corresponding pixel; a removal module, used to remove information of the first fragment in each linked list, the first fragment being located inside the volume data; a volume rendering module, used to perform a mixed volume rendering on multiple volume data slices and the first fragment to obtain a mixed volume rendering result for each pixel in the target image, the multiple volume data slices being obtained by slicing the volume data; an insertion module, used to insert the mixed volume rendering result of each pixel in the target image into the corresponding linked list according to the depth; a color mixing module, used to mix the fragments in the linked list and the mixed volume rendering result for each pixel in the target image to obtain the target image.

[0015] In a third aspect, an embodiment of the present application provides an electronic device, comprising a processor and a computer program stored in a memory and executable on the processor, wherein when the processor executes the computer program, the method for hybrid rendering of surface mesh data and volume data as described in any one of the first aspects above is implemented.

[0016] In a fourth aspect, an embodiment of the present application provides an electronic device, comprising: an X-ray source, a detector, a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for hybrid rendering of surface mesh data and volume data as described in any one of the first aspects above is implemented.

[0017] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the method for hybrid rendering of surface mesh data and volume data described in any one of the first aspects above.

[0018] In a sixth aspect, an embodiment of the present application provides a computer program product. When the computer program product is run on an electronic device, the electronic device executes the mixed rendering method of surface mesh data and volume data described in any one of the above-mentioned first aspects.

[0019] Compared with the prior art, the embodiments of the present application have the following beneficial effects: multiple fragments are obtained by coordinate transformation and rasterization of surface mesh data, the fragments are divided according to their relative positional relationship with the volume data, a first fragment in the volume data and a slice of the volume data are mixed to obtain a mixed volume drawing result, the mixed volume drawing result is then inserted into a linked list of the PPLL according to the depth, and the fragment outside the volume data and the mixed volume drawing result are mixed according to the linked list to obtain a target image as a rendering result, thereby realizing mixed rendering of surface mesh data and volume data, and a semi-transparent effect can be set. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0021] Figure 1 is a structural schematic diagram of an electronic device provided by an embodiment of the present application;

[0022] Figure 2 It is a flowchart of a hybrid rendering method of surface mesh data and volume data provided by an embodiment of the present application;

[0023] Figure 3 It is a schematic diagram of determining the slice direction when the illumination direction and the sight direction are in phase in half-angle slice direct volume rendering;

[0024] Figure 4 It is a schematic diagram of determining the slice direction when the illumination direction and the sight direction are inverted in half-angle slice direct volume rendering;

[0025] Figure 5 It is a schematic diagram of the process of direct volume rendering of half-angle slices;

[0026] Figure 6 is a schematic diagram of PPLL rendering of an original linked list in a specific example of the present application;

[0027] Figure 7 is a schematic diagram of a linked list after the first fragment is removed in a specific example of the present application;

[0028] Figure 8 yes Figure 2 The specific process diagram of S5;

[0029] Fig. 9 is a schematic diagram of a slice rendering result of blending a first fragment into an adjacent volume data slice with a smaller depth in a specific example of the present application;

[0030] Fig.10 is a schematic diagram of a linked list after inserting a volume rendering result in a specific example of the present application;

[0031] Fig.11 It is a structural schematic diagram of a hybrid rendering device of surface mesh data and volume data provided by an embodiment of the present application. DETAILED DESCRIPTION

[0032] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0033] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0034] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0035] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0036] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0037] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0038] The hybrid rendering method of surface mesh data and volume data provided in the embodiment of the present application can be applied to electronic devices, which can specifically include but are not limited to servers, server clusters, workstations, laptop computers, desktop computers and other electronic devices with computing functions. The embodiment of the present application does not impose any restrictions on the specific type of electronic devices.

[0039] Figure 1 FIG. 1 is a block diagram of a partial structure of an electronic device provided in an embodiment of the present application. Figure 1 The electronic device includes: a processor 10, a memory 20, a bus 30, an input device 40, an output device 50, a communication device 60 and a graphics processor 70. The processor 10 and the memory 20 are connected to each other through the bus 30, and the input device 40, the output device 50, and the communication device 60 are also connected to the bus 30. Those skilled in the art will understand that Figure 1 The structure of the electronic device shown in the figure does not constitute a limitation of the electronic device, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0040] Combine the following Figure 1 A detailed introduction to the various components of electronic equipment:

[0041] The processor 10 is the control center of the electronic device, and can run the program stored in the memory 20 to perform various functions and process data. The processor 10 can be a central processing unit (CPU), and the processor 10 can also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In some embodiments, the processor 10 may include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0042] The memory 20 is used to store an operating system, an application program, a boot loader, data, and other programs, such as program codes of computer programs. The memory 20 can also be used to temporarily store data required for executing the program and generated. The memory 20 may include a high-speed random access memory, and may also include a non-volatile memory, such as a flash memory, a hard disk, a multimedia card, a card-type memory, etc. The memory 20 may include a storage unit disposed inside the electronic device, such as a hard disk of the electronic device, and / or a removable external storage unit, such as a mobile hard disk, a USB flash drive, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, etc.

[0043] The input device 40 may include at least one of a keyboard, a mouse, a touch panel, a joystick, etc., and is used to collect user input operations to generate corresponding input signals.

[0044] The output device 50 is used to output information to be provided to the user. The output device 50 generally includes a display, and optionally, a liquid crystal display (LCD), an organic light-emitting diode (OLED), etc. can be used. In addition, the output device can further include a speaker.

[0045] The communication device 60 may include a modem, a network card, etc., for establishing a network connection with other devices and communicating with each other.

[0046] The graphics processor 70 has powerful parallel computing capabilities and is responsible for processing image and graphics related operations.

[0047] The hybrid rendering method of surface mesh data and volume data provided in the embodiment of the present application can be implemented as a computer software program. For example, an embodiment of the present application provides a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 60, and / or installed from a removable external storage unit. When the computer program is executed by the processor 10, the various functions defined in the hybrid rendering method of surface mesh data and volume data provided in the embodiment of the present application are implemented.

[0048] For ease of understanding, the following is a brief introduction to rendering.

[0049] Limited by the fact that the display interface of common display screens is a rectangle in a two-dimensional plane, composed of multiple array-arranged pixels, the display screen cannot fully display all the contents in the three-dimensional model. Since the imaging principle of the human eye and the camera is the same, by simulating a camera placed somewhere in space to shoot the three-dimensional model, the obtained two-dimensional plane image is output to the display screen for display, thereby bringing a realistic effect. This process can also be called rendering.

[0050] Three-dimensional models can be divided into two categories, one is surface mesh data and the other is three-dimensional volume data.

[0051] Since the rendering of the model requires a huge amount of calculation, people have been working to reduce the amount of data that needs to be processed in the rendering process and simplify the data processing process. What the human eye can see is essentially the surface of the object, and it cannot directly see the inside of the object. Therefore, in the process of building a three-dimensional model, only the surface of the object is modeled, and the inside of the object is omitted. At the same time, the surface of the object has different shapes and is difficult to represent uniformly. Therefore, the surface of the object is generally converted into many grids, and a collection of these grids is used to approximate the surface of the object, and the final result is the surface grid data.

[0052] Meshes can also be called primitives, which are the basic units of surface mesh data, including vertices, edges, and faces. Vertices are the basis of meshes, each vertex includes its coordinates in three-dimensional space, edges are line segments connecting two vertices, and faces are plane areas defined by boundaries, usually triangles, but can also be quadrilaterals or other polygons.

[0053] Three-dimensional volume data, also referred to as volume data, generally comes from imaging devices with tomography and reconstruction capabilities, such as magnetic resonance imaging (MRI), computed tomography (CT), positron emission tomography (PET), ultrasound (US), etc. Different from the "hollow" surface grid data, volume data is "solid" and can be understood as a rectangular parallelepiped formed by an array of multiple voxels (corresponding to pixels in a plane image), in which the value of each voxel has its physical meaning. For example, the value of the voxel in the volume data obtained by CT represents the size of the tissue density at the corresponding spatial position.

[0054] Figure 2 A schematic flow chart of a hybrid rendering method of surface mesh data and volume data provided in an embodiment of the present application is shown. As an example but not a limitation, the method can be applied to the above-mentioned electronic device.

[0055] S1: Slice the volume data using slice-based volume rendering to obtain a plurality of volume data slices.

[0056] S1 only needs to be executed before S5, and there is no restriction on the execution order between S2-S4.

[0057] There are two ways to render volume data. One is indirect volume rendering, which extracts surface mesh data from the volume data and then uses the surface mesh data for rendering. The other is direct volume rendering, which is used in this application.

[0058] Direct volume rendering, also known as direct volume rendering, is a method of synthesizing the final rendered image by combining the color of the voxels involved and other metrics obtained by the transfer function according to the path of light propagation and the interaction with each voxel in the volume data under a certain optical model.

[0059] Slice-based direct volume rendering can also be called texture mapping volume rendering. Volume data can be sliced ​​to obtain a series of volume data slices to represent the entire volume data. Volume data slices can also be called proxy geometry, which is a plane polygon and a carrier of volume data. The texture of the volume data slice is obtained by mapping each volume data slice with the volume data as the texture. The volume data slice can be transformed into coordinates to be projected onto the screen, and then rasterized to obtain the rendering result of the volume data slice, also called the rendering result. This process is similar to the process from primitives to fragments in the surface mesh data rendering process. Then the rendering results of each volume data slice are synthesized to obtain the rendering result of the volume data.

[0060] According to the specific slicing method, slicing volume rendering can be divided into orthogonal slicing, line of sight slicing, half-angle slicing, etc., which are not limited here. The following takes half-angle slicing direct volume rendering as an example for explanation.

[0061] The direction of the half-angle slice is determined by the lighting direction and the view direction (i.e. the camera's orientation). When the lighting direction and the view direction are in phase, that is, when the dot product of the two direction vectors is positive, the slice direction (i.e. the normal direction of the resulting proxy geometry) is the half-way vector of the light source direction and the view direction. For example, Figure 3 As shown in the figure, v represents the sight direction, l represents the light direction, and s represents the slice direction. When the illumination direction and the sight direction are opposite, that is, when the dot product of the two direction vectors is negative, the slice direction is the half-way vector in the opposite direction of the light source direction and the sight direction. For example, Figure 4 As shown, v in the figure represents the viewing direction, l represents the light direction, and s represents the slice direction.

[0062] Considering the impact of volume data opacity on lighting, half-angle slice direct volume rendering uses a slice-by-slice iterative rendering method to draw / render volume data slices. Figure 5 As shown, for each volume data slice, the rendering process includes two stages.

[0063] In rendering stage 1, the value of the voxel in the current volume data slice is converted into the color of the voxel through the transfer function, and the coordinates of the slice are converted to the coordinate system to obtain the coordinates of the current volume data slice in the screen coordinate system. The light source buffer is sampled to obtain the color of the illumination (expressed as a normalized pixel value). The color of each voxel is multiplied by the corresponding illumination color to obtain the color of each pixel in the current output image (i.e., the current slice drawing result), and then the camera buffer is updated with the current slice drawing result.

[0064] In rendering stage 2, the color of light attenuated by the current volume data slice is obtained according to the value of the voxel in the current volume data slice and the current light source buffer, and updated to the light source buffer for use in drawing the next volume data slice.

[0065] The drawing process of a single volume data slice can be calculated separately from the blending process of the drawing results of each volume data slice, that is, all the drawing results are blended after the drawing process of all volume data slices is completed. In this case, updating the camera buffer with the current slice drawing result means adding the current slice drawing result to the camera buffer. The storage space occupied by the camera buffer is large because all slice drawing results need to be stored.

[0066] Another situation is that the drawing process of a single volume data slice can be carried out simultaneously with the blending process of the drawing results of each volume data slice, that is, each time the drawing process of a volume data slice is completed, the latest drawing result is blended with all the previously calculated drawing results to obtain a blended result. In this case, updating the camera buffer with the current slice drawing result means taking out the previous blended result from the camera buffer, blending it with the current slice drawing result to obtain the current blended result, and then writing the current blended result to the camera buffer. The storage space occupied by the camera buffer is smaller. After the drawing of the last slice is completed, what is written to the camera buffer is the rendering result of the entire volume data.

[0067] The blending direction of volume data slices can be determined based on the illumination direction and the sight direction. When the dot product of the illumination direction and the sight direction vector is positive, the under operator is used to blend from front to back; when the dot product of the illumination direction and the sight direction vector is negative, the over operator is used to blend from back to front.

[0068] Over operator synthesized from back to front:

[0069] C i =C(s i )a(s i )+(1-a(s i ))C i-1 (1)

[0070] α i =a(s i )+(1-a(s i )) α i-1 (2)

[0071] The under operator synthesized from front to back:

[0072] C i =(1-α i-1 )C(s i )a(s i )+C i-1 (3)

[0073] α i =(1-a i-1 )s(s i )+a i-1 (4)

[0074] Where C represents color, a represents opacity, s represents slice drawing result, and i represents the number of the current volume data slice.

[0075] After executing the above process for all volume data slices, the current mixed result obtained is the rendering result of the volume data.

[0076] This embodiment needs to process the mixed rendering of the three-dimensional model including surface mesh data and volume data. Since two different types of three-dimensional data are involved, the rendering method for a single type of three-dimensional data cannot directly complete the mixed rendering.

[0077] The volume data in the three-dimensional model may be sliced ​​according to a slicing direct volume rendering method (slicing volume rendering for short). The specific method is not limited. In this embodiment, half-angle slicing is used as an example for description.

[0078] S2: coordinate transformation and rasterization are performed on the surface mesh data to obtain multiple fragments.

[0079] The surface mesh data in the 3D model can be obtained by modeling other objects independent of the volume data, or by extracting the surface of an object in the volume data. For example, taking medical image visualization as an example, the surface mesh data of the former can be the modeling data of surgical instruments, and the surface mesh data of the latter can be obtained by extracting the surface of a certain organ in the volume data.

[0080] Each fragment is associated with a pixel in the target image. The surface mesh data can be processed using a rendering pipeline to obtain corresponding multiple fragments, and the specific process and method are not limited.

[0081] The rendering pipeline is generally designed for surface mesh data, and is used to calculate the target image to be output to the display screen based on the set camera parameters and surface mesh data. The display screen corresponds to the imaging plane of the set camera, and is also referred to as the screen in the rendering pipeline.

[0082] In the rendering pipeline, the coordinates of primitives can be transformed from the model's coordinate system to the screen's coordinate system. If the camera's imaging principle is strictly followed, the primitive will lose the depth dimension after this coordinate transformation, changing from three-dimensional to two-dimensional. However, in the subsequent processing steps of the rendering pipeline (such as depth detection and color blending), the depth of the primitive is required, so the depth information of the primitive is not discarded during the conversion, but is retained to be inherited and used by subsequent fragments.

[0083] After the coordinate conversion, the coordinates of the primitives may not be integers. In addition, except for a few special angles (integer multiples of 45°), most straight lines cannot be directly represented by array-arranged pixels. Therefore, they need to be rasterized to adapt to the array-arranged screen and converted into at least one fragment. The color of each fragment can then be calculated based on lighting, texture, etc. The depth here refers to the absolute value of the distance between the primitive / fragment and the screen, which may be z or -z according to the definition of the graphics interface.

[0084] The size of a fragment is the same as the pixel in the target image as the rendering result. Both contain their own coordinates and color (expressed in pixel values). In addition, the fragment also contains other information, such as depth, normal, texture, etc. Fragments and pixels with the same coordinates can be said to correspond to each other or be related to each other. A fragment is only associated with one pixel, but a pixel may be associated with multiple fragments.

[0085] If there is only surface mesh data, the color of a single pixel in the target image can be obtained by performing fragment processing (such as depth detection, color blending, etc.) on the fragment associated with the pixel. The target image can be obtained after executing the above process for each pixel, but it is not applicable to the situation where mixed rendering is required in this embodiment.

[0086] S3: Generate a linked list of each pixel in the target image.

[0087] The linked list includes the information of the fragments associated with the corresponding pixels. The linked list is used for the fragment processing part of the semi-transparent mesh rendering. The specific methods of semi-transparent mesh rendering can be pixel-level linked list (Per-Pixel Linked List, PPLL), adaptive transparency (Adaptive Transparency, AT), depth peeling (Depth peeling), dual depth peeling (Dualdepth peeling), weighted blending (Weighted, blended), etc. The following takes PPLL as an example for explanation.

[0088] PPLL builds and maintains a linked list ListBuffer for each pixel in the target image. The linked list of each pixel stores the information of all the fragments corresponding to the pixel. The information of the fragments in the linked list includes the color, depth and index of the fragment. The index is used to indicate the position of the last fragment drawn for the pixel in the linked list. PPLL also builds and maintains a texture image (Head Image) of the same size as the target image. Each pixel in the texture image is used to store the position of the last fragment drawn for the pixel in the linked list.

[0089] After rasterization, the primitives to be displayed are converted into multiple fragments, and the information of these fragments is filled into the linked list of associated pixels. The fragments in the linked list of each pixel are sorted according to the depth to determine the index information. A single fragment is actually obtained by coordinate conversion and rasterization of part or all of the primitives, so the depth of the fragment is actually the depth of the part or all of the primitives from which it comes. For example, if the fragments are arranged in order from small to small depth, then the fragment B pointed to by the index value of fragment A is the adjacent fragment with a greater depth of fragment A.

[0090] For example, Figure 6As shown, a pixel is associated with 4 fragments: fragment0, fragment1, fragment3, and fragment5. The depth order is fragment0>fragment1>fragment3>fragment5. The index value of fragment5 is fragment3, the index value of fragment3 is fragment1, and the index value of fragment1 is fragment0.

[0091] S4: Remove the information of the first fragment in each linked list.

[0092] The first fragment is located inside the volume data, which means that part or all of the primitives from which it is derived are located inside the cuboid occupied by the volume data. The first fragment participates in the rendering of the volume data but does not directly perform PPLL rendering, so it is removed from the linked list.

[0093] In a hybrid rendering scene, volume data generally occupies most of the three-dimensional space. Therefore, in order to simplify the judgment process of the first fragment, it is possible to judge whether the first fragment exists only based on the depth. For each linked list, it is possible to judge whether the first fragment exists based on the depth of each fragment therein. Specifically, the depth from the vertex of the cuboid occupied by the volume data to the screen is obtained, and the range defined by the minimum and maximum values ​​is the depth range of the volume data. If the depth of a fragment belongs to the depth range, the fragment is located within the volume data and belongs to the first fragment; if the depth of a fragment does not belong to the depth range, the fragment is located outside the volume data and belongs to the second fragment.

[0094] When the first fragment exists, the information of the first fragment is removed from the linked list, and the relevant index value is adaptively updated. The relevant index value here only includes the index value that was originally the first fragment. Specifically, the index value that was originally the first fragment can be modified to the second fragment in the original index value of the first fragment.

[0095] Still continuing Figure 6 For example, according to the depth, we know that fragment1 and fragment3 are located inside the volume data and belong to the first fragment. Remove fragment1 and fragment3 from the linked list. After removing the first fragment, the index value of fragment5 among the remaining fragments was originally the first fragment fragment3, which is the relevant index value that needs to be updated. The original index values ​​of fragment1 and fragment3 are fragment0 and fragment1 respectively, where fragment0 is the second fragment outside the volume data. Change the index value of fragment5 to fragment0. After removing the first fragment, the linked list is as follows Figure 7 shown.

[0096] Optionally, a mask image of the same size can be constructed and maintained for the volume data. The mask image will add a channel to mark the voxel corresponding to the first fragment. Specifically, the value of the corresponding voxel in the mask image corresponding to the voxel of the first fragment can be modified to the number of the first fragment. Other channels of the mask can mark the type of voxel. For example, in medical visualization, other channels of the mask can be used to mark whether the voxel belongs to human tissue.

[0097] S5: Perform mixed volume rendering on the plurality of volume data slices and the first fragment to obtain a mixed volume rendering result of each pixel in the target image.

[0098] like Figure 8 As shown, in a specific embodiment of the present application, S5 includes the following parts.

[0099] S51: Sort the volume data slices and the first fragment according to depth to determine a volume data slice adjacent to the first fragment in depth.

[0100] The depth of a volume data slice refers to the distance between the volume data slice and the screen. The volume data slice can be understood as a plane. If it is parallel to the screen, then the distance is the depth of the volume data slice. If it is not parallel to the screen, then the distance from a certain point in the volume data slice (such as the midpoint or a vertex) to the screen can be uniformly defined as the depth of the volume data slice.

[0101] The depths of different volume data slices are generally different. The volume data slices and the first fragment are sorted according to depth. If a volume data slice has the same depth as the first fragment, then the volume data slice is the volume data slice adjacent to the first fragment in depth. If there is no volume data slice with the same depth as the first fragment, then the volume data slice with a depth less than and closest to the first fragment (if any) and the volume data slice with a depth greater than and closest to the first fragment (if any) are the volume data slices adjacent to the first fragment in depth.

[0102] S52: Drawing the volume data slices in sequence to obtain slice drawing results of the volume data slices.

[0103] The execution order between S51 and S52 is only for illustration and is not actually limited. For example, when the drawing process of a single volume data slice and the mixing process of the drawing results of each volume data slice are performed simultaneously, it can be checked whether there is a first fragment between the current volume data slices after the current volume data slice is drawn. If so, the first fragment is mixed into the slice drawing result of the current volume data slice, and then mixed with the previous mixed result to obtain the current mixed result.

[0104] The existing slice drawing method can be used to draw the volume data slices, and there is no restriction here. The light source used in the slice drawing process is essentially a lighting model used to describe the lighting in the three-dimensional model. There is no restriction on its type. For example, a lighting model with real-time and no need for pre-calculation, movable light sources, soft shadows, hard shadows, specular highlights, multiple scattering, color bleeding, ambient occlusion and other advanced effects can be used to obtain a more realistic and real-time rendering effect.

[0105] S53: Blending the first fragment into a slice rendering result of an adjacent volume data slice.

[0106] If the first fragment has only one adjacent volume data slice, for example, the depth of the first fragment is the same as the depth of a certain volume data slice, or the first fragment is located on the same side of all volume data slices in depth, the slice drawing result of the volume data slice is directly used as the mixing object of the first fragment. If the first fragment has two adjacent volume data slices, the mixing object of the first fragment can be determined according to the iteration order in the process of mixing the slice drawing results. Specifically, the first fragment can be mixed into the slice drawing result of the volume data slice that is adjacent in depth and has the first iteration order in the process of mixing the slice drawing results. For example, if the mixed result of each slice drawing result is iteratively calculated in the order of depth from small to large, the first fragment can be mixed into the slice drawing result of the adjacent volume data slice with a smaller depth.

[0107] For example, Fig. 9 As shown, in volume rendering, the mixed results of the drawing results of each slice are iteratively calculated in order from small to large depth, corresponding to the left to right order in the figure. The fragments that make up the circle are located between the two slices, and the fragments that make up the triangle are located to the right of the right slice. Therefore, the fragments that make up the circle are mixed into the left slice, and the fragments that make up the triangle are mixed into the right slice.

[0108] The volume data slice itself is generally very large, and its slice drawing result is generally associated with multiple pixels in the target image, or even the entire target image, while a single first slice is only associated with one pixel. The mixing result of the pixels at the corresponding positions (i.e., the same coordinates) in the first slice and its mixed object can be calculated according to the mixing formula of the slice volume drawing. The mixing formula involves color and opacity. The color and opacity of the pixels at the corresponding positions in the mixed object of the first slice are replaced with the color and opacity of the mixing result, thereby completing the mixing of the first slice and the slice drawing result as its mixed object.

[0109] Optionally, if the dynamic ranges of the PPLL rendering and the slice volume rendering are different, dynamic range adjustment can be performed before blending to unify the dynamic ranges.

[0110] S54: Mix the rendering results of each slice to obtain a mixed volume rendering result of each pixel in the target image.

[0111] The blending direction can be determined according to the illumination direction and the sight direction, and the slice drawing results mixed with the first fragment are blended according to the blending direction to obtain a blended body drawing result. The blending of the slice drawing results results in an intermediate image of the same size as the target image, in which the information of each pixel (including color, opacity, depth, etc.) is the blended body drawing result of the corresponding pixel in the target image.

[0112] A pixel in the target image substantially corresponds to a line in the volume data, and the depth of the mixed volume rendering result of the pixel can be determined based on the depth of the line, for example, it can be the distance from the midpoint of the line to the screen; or, the depth of the mixed volume rendering results of all pixels can be uniformly set to the same value, for example, it can be the distance from the center point of the cuboid of the volume data to the screen.

[0113] S6: Insert the mixed body drawing result of each pixel in the target image into the corresponding linked list according to the depth.

[0114] For each pixel in the target image, the mixed body rendering result and the fragments in the corresponding linked list can be sorted according to depth to obtain the depth sorting result, and then the mixed body rendering result can be inserted into the corresponding linked list according to the depth sorting result, and the relevant index value can be adaptively updated.

[0115] No matter how the depth of the mixed volume rendering result is set, it will fall within the depth range of the volume data, and the depth is often used to determine the first fragment. Therefore, generally speaking, the mixed volume rendering result will occupy the position of the original first fragment.

[0116] The relevant index values ​​here include the index value modified in the process of removing the first fragment and the index value of the mixed body rendering result. Specifically, the index value modified in the process of removing the first fragment can be modified from the second fragment to the mixed body rendering result, and the index value of the mixed body rendering result can be modified to the second fragment in the original index value of the first fragment.

[0117] Still continuing Figure 7 For example, the depth of the mixed body rendering result is between fragment0 and fragment5. The mixed body rendering result is inserted into the linked list, and the index value of fragment5 is changed from fragment0 to the mixed body rendering result. The index value of the mixed body rendering result is fragment0. The linked list after inserting the mixed body rendering result is as follows Fig.10 As shown, for the convenience of display, the mixed volume rendering result in the figure is abbreviated as volume rendering result.

[0118] S7: For each pixel in the target image, the fragments in the linked list and the mixed body drawing results are mixed to obtain the target image.

[0119] The fragment and mixed body drawing results in the linked list are already sorted by depth, and can be mixed in sequence starting from the head of the linked list.

[0120] The color mixing formula combines the color mixing formula of the fragment itself of PPLL and the color mixing formula of different slice drawing results in volume rendering. For example, assuming that the linked list is sorted in ascending order, the color mixing can be performed according to the following formula:

[0121] a i+1 =a i +(1-a i )*b i+1 (5)

[0122] C i+1 =C i +S i+1 *b i+1 *(1-a i ) (6)

[0123] C i+1 =C i +H i+1 (1-a i ) (7)

[0124] Where a (initial value is 0) represents the opacity of the final output, b represents the opacity of the fragment / mixed body drawing result, C (initial value is 0) represents the color of the final output, S represents the color of the fragment, and H represents the color of the mixed body drawing result. That is, for the fragment, use formulas (5) and (6) for color mixing; for the mixed body drawing result, use formulas (5) and (7) for color mixing.

[0125] For a single pixel, use the above formula to mix all the fragments and mixed body drawing results in its linked list to obtain the color of the pixel; perform the above process for each pixel in the target image to obtain the target image for display.

[0126] Optionally, if the dynamic ranges of the PPLL rendering and the slice volume rendering are different, dynamic range adjustment can be performed before blending to unify the dynamic ranges.

[0127] Through the implementation of this embodiment, the surface mesh data is coordinate transformed and rasterized to obtain multiple fragments, the fragments are divided according to their relative position relationship with the volume data, the first fragment in the volume data and the volume data slice are mixed to obtain a mixed volume rendering result, and then the mixed volume rendering result is inserted into the linked list of PPLL according to the depth, and the fragment outside the volume data and the mixed volume rendering result are mixed according to the linked list to obtain a target image as a rendering result, thereby realizing mixed rendering of surface mesh data and volume data, and a semi-transparent effect can be set.

[0128] Fig.11 A schematic diagram of the structure of a hybrid rendering device for surface mesh data and volume data provided in an embodiment of the present application is shown, the device comprising a rendering module 101, a generation module 102, a removal module 103, a volume rendering module 104, an insertion module 105 and a color mixing module 106.

[0129] The rendering module 101 is used to perform coordinate conversion and rasterization on the surface mesh data to obtain a plurality of fragments, each of which is associated with a pixel in the target image.

[0130] The generation module 102 is used to generate a linked list of each pixel in the target image, where the linked list includes information of fragments associated with the corresponding pixels.

[0131] The removal module 103 is used to remove the information of the first fragment in each linked list, where the first fragment is located inside the volume data.

[0132] The volume rendering module 104 is configured to perform mixed volume rendering on the plurality of volume data slices and the first fragment to obtain a mixed volume rendering result of each pixel in the target image. The plurality of volume data slices are obtained by slicing the volume data.

[0133] The inserting module 105 is used to insert the mixed volume rendering result of each pixel in the target image into the corresponding linked list according to the depth.

[0134] The color mixing module 106 is used to mix the fragments in the linked list and the mixed body rendering results for each pixel in the target image to obtain the target image.

[0135] Optionally, the hybrid volume rendering module 104 includes a sorting unit, a rendering unit, a first blending unit, and a second blending unit. The sorting unit is used to sort the volume data slices and the first fragment according to depth to determine the volume data slices adjacent to the first fragment in depth. The rendering unit is used to render the volume data slices in sequence to obtain slice rendering results of the volume data slices. The first blending unit is used to blend the first fragment into the slice rendering result of an adjacent volume data slice. The second blending unit is used to blend the slice rendering results to obtain a hybrid volume rendering result for each pixel in the target image.

[0136] Optionally, the first blending unit is specifically configured to blend the first fragment into a slice rendering result of an adjacent volume data slice that has a preceding iteration order in a process of blending the slice rendering results.

[0137] Optionally, the insertion module 105 is specifically used to sort the mixed body rendering result and the fragments in the corresponding linked list according to depth for each pixel in the target image to obtain a depth sorting result; add the mixed body rendering result to the corresponding linked list according to the depth sorting result, and adaptively update the relevant index value.

[0138] Optionally, the removal module 103 is specifically used to determine whether the first fragment exists in each linked list according to the depth of each fragment therein; when the first fragment exists, remove the information of the first fragment from the linked list and adaptively update the relevant index value.

[0139] Optionally, a slicing module (not shown in the figure) is further included, which is used to slice the volume data using half-angle slicing volume rendering to obtain multiple volume data slices.

[0140] Optionally, a mask maintenance module (not shown) is also included, which is used to construct a mask image of the same size as the volume data; and modify the value of the corresponding voxel of the voxel corresponding to the first fragment in the mask image to the number of the first fragment.

[0141] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / modules / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0142] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0143] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0144] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0145] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned various method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device that can carry the computer program code to the camera / electronic device, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, RandomAccess Memory), electric carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0146] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0147] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0148] In the embodiments provided in the present application, it should be understood that the disclosed devices / network equipment and methods can be implemented in other ways. For example, the device / network equipment embodiments described above are merely schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0149] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0150] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A hybrid rendering method of surface mesh data and volume data, characterized in that: The method comprises: Performing coordinate transformation and rasterization on the surface mesh data to obtain a plurality of fragments, each of the fragments being associated with a pixel in the target image; Generate a linked list of each pixel in the target image, wherein the linked list includes information of the fragment associated with the corresponding pixel; Remove information of the first fragment in each of the linked lists, where the first fragment is located inside the volume data; Performing mixed volume rendering on a plurality of volume data slices and the first fragment to obtain a mixed volume rendering result of each pixel in the target image, wherein the plurality of volume data slices are obtained by slicing the volume data; Inserting the mixed body rendering result of each pixel in the target image into the corresponding linked list according to the depth; For each pixel in the target image, the fragments in the linked list and the mixed body drawing result are mixed to obtain the target image.

2. The method according to claim 1, characterized in that The step of performing mixed volume rendering on the plurality of volume data slices and the first fragment to obtain a mixed volume rendering result of each pixel in the target image includes: sorting the volume data slices and the first fragment according to depth to determine the volume data slices adjacent to the first fragment in depth; Drawing the volume data slice to obtain a slice drawing result of the volume data slice; blending the first fragment into a slice rendering result of an adjacent slice of the volume data; The slice rendering results are mixed to obtain a mixed body rendering result of each pixel in the target image.

3. The method according to claim 2, characterized in that The step of blending the first fragment into the slice rendering result of an adjacent slice of the volume data comprises: The first fragment is blended into a slice rendering result of a slice of the volume data that is adjacent in depth and precedes in an iteration order in the blending of the slice rendering results.

4. The method according to claim 1, characterized in that Inserting the mixed body rendering result of each pixel in the target image into the corresponding linked list according to the depth includes: For each pixel in the target image, sorting the mixed volume rendering result and the corresponding fragments in the linked list according to depth to obtain a depth sorting result; The mixed volume rendering result is inserted into the corresponding linked list according to the depth sorting result.

5. The method according to claim 1, characterized in that The information of removing the first fragment in each of the linked lists includes: For each of the linked lists, judging whether the first fragment exists according to the depth of each of the fragments therein; When the first fragment exists, the information of the first fragment is removed from the linked list, and the relevant index value is adaptively updated.

6. The method according to any one of claims 1 to 5, characterized in that: Before performing mixed volume rendering on the plurality of volume data slices and the first fragment, the method further includes: The volume data is sliced ​​using half-angle slice volume rendering to obtain the plurality of volume data slices.

7. The method according to any one of claims 1 to 5, characterized in that: The method further comprises: constructing a mask image of the same size as the volume data; The value of the corresponding voxel of the voxel corresponding to the first fragment in the mask image is modified to the number of the first fragment.

8. A hybrid rendering device for surface mesh data and volume data, characterized in that: The device comprises: A rendering module, used for performing coordinate conversion and rasterization on the surface mesh data to obtain a plurality of fragments, each of which is associated with a pixel in the target image; A generating module, used for generating a linked list of each pixel in the target image, wherein the linked list includes information of the fragment associated with the corresponding pixel; A removing module, used to remove information of the first fragment in each of the linked lists, where the first fragment is located inside the volume data; a hybrid volume rendering module, configured to perform hybrid volume rendering on a plurality of volume data slices and the first fragment to obtain a hybrid volume rendering result of each pixel in the target image, wherein the plurality of volume data slices are obtained by slicing the volume data; An insertion module, used for inserting the mixed body rendering result of each pixel in the target image into the corresponding linked list according to the depth; A color mixing module is used to mix the fragments in the linked list and the mixed body drawing result for each pixel in the target image to obtain the target image.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable by the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images