Method, apparatus, electronic device, and storage medium for determining transfer function of volume rendering

Abstract

The present invention discloses a method, apparatus, electronic device, and storage medium for determining a transfer function of volume rendering. The method includes obtaining an original image and a segmented image of the original image, and respectively determining an original block image of the original image and a segmented block image of the segmented image; determining a spliced image based on the original block image and the segmented block image, and inputting the spliced image into a pre-trained transfer parameter determination model to obtain transfer parameters output by the model; obtaining array parameters corresponding to the transfer parameters, and determining a transfer function based on the transfer parameters and the array parameters; performing volume rendering on the original image using the transfer function; solving the problems in the prior art that it is time-consuming to obtain the transfer function through a modulation method and the accuracy is relatively low, and achieving the reduction of the fine-tuning time of the transfer function and the improvement of the accuracy of the transfer function.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular, to a method, apparatus, electronic device, and storage medium for determining a transfer function of volume rendering. Background Art

[0002] When performing nodular target reconstruction, a doctor needs to finely manually adjust the transfer function according to the nodular characteristics in a specific rendering area. Generally, this manual adjustment can ensure a good volume rendering effect for target reconstruction. However, fine-tuning the transfer function is time-consuming and laborious. Even an experienced doctor needs more than ten minutes to obtain a good transfer function. If the doctor lacks experience, it is very likely that a transfer function with a good effect cannot be adjusted, affecting the diagnosis result. Therefore, it is time-consuming and has low accuracy to obtain the transfer function based on the modulation method of the prior art. Summary of the Invention

[0003] The present invention provides a method, apparatus, electronic device, and storage medium for determining a transfer function of volume rendering, so as to solve the problems that it is time-consuming and has low accuracy to obtain the transfer function by the modulation method of the prior art, and realizes shortening the fine-tuning time of the transfer function and improving the accuracy of the transfer function.

[0004] In a first aspect, an embodiment of the present invention provides a method for determining a transfer function of volume rendering, and the method includes:

[0005] Obtain an original image and a segmented image of the original image, and respectively determine an original cut image of the original image and a segmented cut image of the segmented image;

[0006] Determine a spliced image based on the original cut image and the segmented cut image, and input the spliced image into a pre-trained transfer parameter determination model to obtain transfer parameters output by the model;

[0007] Obtain an array parameter corresponding to the transfer parameter, and determine a transfer function based on the transfer parameter and the array parameter; the transfer function performs volume rendering on the original image.

[0008] Optionally, the step of respectively determining the original cut image of the original image and the segmented cut image of the segmented image includes:

[0009] Determine a target object in the segmented image, and determine a cut starting point based on the target object;

[0010] Obtain a preset cut size, and perform cut processing on the original image and the segmented image respectively based on the cut starting point and the cut size to obtain the original cut image of the original image and the segmented cut image of the segmented image.

[0011] Optionally, determining the stitched image based on the original cut image and the segmented cut image includes:

[0012] Obtain the pixel values of each pixel point in the segmented cut image, perform binarization processing on each of the pixel values based on at least one preset pixel threshold, and perform category marking on each pixel point in the segmented cut image based on the binarization processing result to obtain a segmented cut processed image including at least one category;

[0013] Perform image normalization processing on the original cut image to obtain the original cut processed image of the original cut image;

[0014] Perform stitching processing on the segmented cut processed image and the original cut processed image to obtain the stitched image.

[0015] Optionally, performing stitching processing on the segmented cut processed image and the original cut processed image to obtain the stitched image includes:

[0016] Obtain the channel data of the original cut processed image as the first channel data, and obtain the channel data of the segmented cut processed image as the second channel data;

[0017] Perform channel stitching on the first channel data and the second channel data to obtain a stitched image including two channels.

[0018] Optionally, the training of the transfer parameter determination model includes:

[0019] Obtain the labeled sample image and the labeled transfer parameters for training the transfer parameter determination model;

[0020] Perform image processing on the labeled sample image to obtain a sample stitched image, and input the sample stitched image into the transfer parameter determination model to be trained to obtain the training transfer parameters output by the model;

[0021] Determine at least one parameter loss function of the transfer parameter determination model in the current iteration based on the training transfer parameters and the labeled transfer parameters, and determine the model loss function of the transfer parameter determination model in the current iteration based on each of the parameter loss functions and the parameter weights respectively corresponding to each parameter loss function; the weight values of each of the parameter weights are different;

[0022] Adjust the model parameters of the transfer parameter determination model in the current iteration based on the model loss function, and continue iterative training based on the model after parameter adjustment until the iterative stop condition is met and then stop training to obtain the trained transfer parameter determination model.

[0023] Optionally, the parameter loss function includes a parameter mean squared error loss function, a parameter distance loss function, and a parameter mapping loss function generated based on the parameters at corresponding positions in the training transfer parameters and the annotation transfer parameters.

[0024] Optionally, the method further includes:

[0025] Obtain a pre-trained volume rendering model;

[0026] Input the original image and the transfer function into the volume rendering model to obtain a completed 3D effect image.

[0027] In a second aspect, an embodiment of the present invention further provides a device for determining a transfer function of volume rendering. The device includes:

[0028] An image acquisition module, configured to acquire an original image and a segmented image of the original image, and respectively determine an original cut image of the original image and a segmented cut image of the segmented image;

[0029] A transfer parameter determination module, configured to determine a spliced image based on the original cut image and the segmented cut image, and input the spliced image into a pre-trained transfer parameter determination model to obtain transfer parameters output by the model;

[0030] A transfer function determination module, configured to obtain an array parameter corresponding to the transfer parameter, and determine a transfer function based on the transfer parameter and the array parameter; the transfer function performs volume rendering on the original image.

[0031] In a third aspect, an embodiment of the present invention further provides an electronic device, including:

[0032] At least one processor; and

[0033] A memory communicatively connected to the at least one processor; wherein,

[0034] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the method for determining the transfer function of volume rendering according to any embodiment of the present invention.

[0035] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium storing computer instructions for causing a processor to execute the method for determining the transfer function of volume rendering according to any embodiment of the present invention when executed.

[0036] The technical solution of the embodiment of the present invention obtains the original image and the segmented image of the original image, and respectively determines the original cut image of the original image and the segmented cut image of the segmented image; determines the spliced image based on the original cut image and the segmented cut image, and inputs the spliced image into the pre-trained transfer parameter determination model to obtain the transfer parameters output by the model; obtains the array parameters corresponding to the transfer parameters, and determines the transfer function based on the transfer parameters and the array parameters; the transfer function performs volume rendering on the original image, solving the problems that the existing technology is time-consuming and has low accuracy in obtaining the transfer function by modulation method, realizing shortening the fine-tuning time of the transfer function and improving the accuracy of the transfer function.

[0037] It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of the present invention, nor is it used to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0039] Figure 1 is a flowchart of a method for determining a transfer function for volume rendering according to an embodiment of the present invention;

[0040] Figure 2 is a flowchart of another method for determining a transfer function for volume rendering according to an embodiment of the present invention;

[0041] Figure 3 is a schematic structural diagram of a device for determining a transfer function for volume rendering according to an embodiment of the present invention;

[0042] Figure 4 is a schematic structural diagram of an electronic device for implementing the method for determining a transfer function for volume rendering according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0043] In order to enable those skilled in the art to better understand the solution of the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0044] It should be noted that the terms "first", "second", etc. in the description, claims and the above drawings of the present invention are used to distinguish similar objects, and do not necessarily have to be used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein.

[0045] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only for illustrative purposes, and are not used to limit the scope of these messages or information.

[0046] It can be understood that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, usage scopes, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0047] For example, when responding to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested by the user will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, an application program, a server, or a storage medium that executes the operations of the technical solutions of the present disclosure according to the prompt message.

[0048] As an optional but non-limiting implementation manner, the manner of sending a prompt message to the user in response to receiving an active request from the user may be, for example, in the form of a pop-up window, and the prompt message may be presented in text in the pop-up window. In addition, the pop-up window may also carry a selection control for the user to select "agree" or "disagree" to provide personal information to the electronic device.

[0049] It can be understood that the above process of notifying and obtaining the user's authorization is only illustrative and does not constitute a limitation on the implementation manner of the present disclosure. Other manners that meet relevant laws and regulations can also be applied to the implementation manner of the present disclosure.

[0050] It can be understood that the data involved in the technical solution (including but not limited to the data itself, the acquisition or use of the data) should comply with the requirements of the corresponding laws, regulations and related provisions.

[0051] Figure 1 This is a flowchart of a method for determining a transfer function for volume rendering provided by an embodiment of the present invention. This embodiment is applicable to the case of performing volume rendering on three-dimensional image data. This method can be executed by a device for determining a transfer function for volume rendering. The device for determining a transfer function for volume rendering can be implemented in the form of hardware and / or software, and the device for determining a transfer function for volume rendering can be configured in an intelligent terminal and a cloud server. As Figure 1 shown, the method includes:

[0052] S110. Obtain the original image and the segmented image of the original image, and respectively determine the original cut image of the original image and the segmented cut image of the segmented image.

[0053] In the embodiment of the present invention, the original image can be understood as the image to be drawn into a 3D effect, that is, the image data of the original image is three-dimensional image data. Optionally, the image type of the original image can be any type, such as a medical image, a portrait image or a landscape image. For the sake of convenience of introduction, in this embodiment, the original image is taken as an example of three-dimensional medical image data for introduction. It should be noted that the above-mentioned image types of the original image are only exemplary introductions of several optional types, and the image type of this embodiment can also be other types, and this embodiment does not limit this. Correspondingly, the image type of the subsequent image obtained based on the original image is also not limited.

[0054] In this embodiment, when the original image is a three-dimensional medical image, the three-dimensional medical image corresponding to the preset part of a person or an animal can be directly obtained based on a medical imaging device. For example, a CT scan of the human lung is performed to obtain a CT image corresponding to the human lung. Of course, depending on the different medical imaging devices, the obtained three-dimensional medical images are also different, and this embodiment does not limit the specific medical imaging device. Optionally, the method for obtaining the original image can also be to read the image data from a database stored in the device and render the image data to obtain the original image. This embodiment does not limit the method for obtaining the original image.

[0055] Specifically, on the basis of obtaining the original image, the segmented image of the original image is obtained based on the original image. Optionally, obtaining the segmented image of the original image can be to process the original image based on a traditional segmentation algorithm to obtain the corresponding segmented image of the original, or to input the original image into a pre-trained segmented image model to obtain the segmented image corresponding to the original image.

[0056] Specifically, on the basis of the above embodiment, the original image and the segmented image are respectively subjected to a cutting process to obtain the original cut image of the original image and the segmented cut image of the segmented image.

[0057] Among them, chunking can be understood as a processing method for extracting the region of interest in an image. For example, if the image contains a target object, chunking the image based on the target object can obtain a chunked image including the target object, and subsequent processing can be performed on the chunked image. The effect of chunking the image and performing subsequent processing on the obtained chunked image is that the accuracy of subsequent image processing can be ensured based on the target object in the chunked image, and removing the edge region can reduce the data calculation amount of subsequent image processing, thereby improving the processing efficiency of image processing.

[0058] Optionally, in this embodiment, the method for respectively determining the original chunked image of the original image and the segmented chunked image of the segmented image may include: determining the target object in the segmented image, and determining the chunking starting point based on the target object; obtaining a preset chunking size, and respectively chunking the original image and the segmented image based on the chunking starting point and the chunking size to obtain the original chunked image of the original image and the segmented chunked image of the segmented image.

[0059] For the segmented image, the technical solution of this embodiment can select multiple objects included in the segmented image to obtain the target object in the segmented image. Since the segmented image is obtained by performing segmentation processing on the original image, the target objects included in the original image and the segmented image are the same. Therefore, the target object selected based on the segmented image is also used as the target object in the original image.

[0060] In this embodiment, an object can be understood as a foreign object located at the imaging part, such as a pulmonary nodule located in the lung. Optionally, since different objects have different density information, the pixel values of the pixel points of the pulmonary nodule and the normal part of the lung in the CT image are different. Since there may be multiple nodules in the lung, in order to make the subsequent processing results more accurate, the nodules in the lung can be screened to select the nodules that meet the conditions as the objects for subsequent processing. Optionally, the method for screening the nodules in the lung to obtain the target nodules that meet the conditions may include screening based on the nodule size and nodule type of each nodule. For example, any nodule with a nodule size greater than 3 mm and a nodule type of ground-glass nodule, solid nodule, calcified nodule, or interstitial nodule of the lobe is used as a nodule that meets the conditions.

[0061] It should be noted that the above method for determining the target object is only used as an exemplary introduction method and is not a limitation on this embodiment.

[0062] Optionally, the position of the target object in the segmented image can be determined based on the pixel values of the pixel points in the segmented image, that is, determining the number of voxels occupied by the pulmonary nodule in the segmented image. Correspondingly, the position of the target object in the original image can also be determined accordingly.

[0063] In this embodiment, assuming that the density of the target object is uniform, the centroid position of the target object in the image can be determined based on the number of voxels occupied by the target object and the position of the voxel in the image. The position of the centroid is used as the starting point of the image cutting process. The purpose is to place the target object at the center of the cut image or to avoid the target object being located at the edge of the cut image as much as possible, thereby avoiding the problem of inaccurate image processing results caused by subsequent cutting processing and ensuring the accuracy of subsequent image processing.

[0064] Specifically, obtain the cutting size of the image cutting process. Optionally, the size varies according to the shape of the cut. For example, if the cut shape is circular, the obtained cutting size is the cutting radius; if the cut shape is quadrilateral, the obtained cutting size is the cutting length and the cutting width. There is no limitation on this. It should be noted that the cutting size of the cut image can be determined based on the image size of the original image and the size of the target object, and no specific limitation is provided here.

[0065] Based on the above embodiment, the technical solution of this embodiment can perform cutting processing on the original image based on the cutting starting point and the cutting size to obtain the original cut image of the original image, and perform cutting processing on the segmented image based on the cutting starting point and the cutting size to obtain the segmented cut image of the segmented image.

[0066] S120. Determine a spliced image based on the original cut image and the segmented cut image, and input the spliced image into a pre-trained transfer parameter determination model to obtain the transfer parameters output by the model.

[0067] In the embodiment of the present invention, in order to accurately obtain the transfer parameters of the transfer function, a transfer parameter determination model for determining transfer parameters is pre-trained. Optionally, on the basis of obtaining the original cut image and the segmented cut image, in order to match the data channels of the input data of the model, it is necessary to pre-splice the original cut image and the segmented cut image to obtain a spliced image. And determine the transfer parameters based on the spliced image.

[0068] Optionally, the method for determining the spliced image based on the original cut image and the segmented cut image in this embodiment may include: obtaining the pixel values of each pixel point in the segmented cut image, performing binarization processing on each pixel value based on at least one preset pixel threshold, and performing category marking on each pixel point in the segmented cut image based on the binarization processing result to obtain a segmented cut processing image including at least one category; performing image normalization processing on the original cut image to obtain an original cut processing image of the original cut image; splicing the segmented cut processing image and the original cut processing image to obtain a spliced image.

[0069] Specifically, taking the segmented and cut image of the blood vessels and nodules in the lungs as an example. Since some nodules are attached to the surface of the blood vessels, for such nodules, their density is neither the same as that of ordinary nodules nor the same as that of blood vessels. Therefore, the technical solution of this embodiment can preset multiple pixel thresholds, such as the image background threshold, the nodule threshold, the blood vessel threshold, and the blood vessel threshold of the attached nodules. Based on the above four thresholds, the segmented and cut image is binarized to obtain a binarized result with four classifications. Specifically, based on the binarized result, each pixel point is classified and pixel mapping processing is performed to obtain a segmented and cut processed image containing at least one category. For example, in the obtained segmented and cut image, 0 is the background, 1 is the blood vessel segmentation, 2 is the nodule segmentation, and 3 is the intersection area of the nodule and the blood vessel.

[0070] Specifically, the original cut image is binarized. For example, the original cut image is normalized according to a window level of -400 and a window width of 1000 to obtain an original cut processed image in which most of the values in the image are between -1 and 1.

[0071] Specifically, on the basis of obtaining the segmented and cut processed image and the original cut processed image, the two images are spliced in the channel dimension to obtain a spliced image that meets the requirements of the model input data.

[0072] Specifically, in this embodiment, the method of splicing the segmented and cut processed image and the original cut processed image to obtain a spliced image may include obtaining the channel data of the original cut processed image as the first channel data, and obtaining the channel data of the segmented and cut processed image as the second channel data; splicing the first channel data and the second channel data in channels to obtain a spliced image containing two channels.

[0073] Optionally, the spliced image obtained based on the above embodiment is input into a pre-trained transfer parameter determination model to obtain the transfer parameters output by the model. Among them, the transfer parameters may include four parameters, such as (hu1, hu2, hu3, hu4). The transfer parameter determination model may be a model obtained by combining a ResNet network and a linear layer; optionally, the ResNet network may also be replaced by network structures such as ResUNet or InceptionNet, and this is not limited.

[0074] On the basis of the above embodiment, the technical solution of this embodiment further includes training the transfer parameter determination model before determining the transfer parameters to obtain a trained transfer parameter determination model.

[0075] Optionally, determining the training of the model by passing parameters includes: obtaining labeled sample images and labeled passing parameters for training the passing parameter determination model; performing image processing on the labeled sample images to obtain sample stitching images, and inputting the sample stitching images into the passing parameter determination model to be trained to obtain the training passing parameters output by the model; determining at least one parameter loss function of the passing parameter determination model in the current iteration based on the training passing parameters and the labeled passing parameters, and determining the model loss function of the passing parameter determination model in the current iteration based on each parameter loss function and the parameter weights respectively corresponding to their respective parameter loss functions; the weight values of the parameter weights are different; adjusting the model parameters of the passing parameter determination model in the current iteration based on the model loss function, and continuing the iterative training based on the model after parameter adjustment until the training stops when the iteration stop condition is met, and obtaining the trained passing parameter determination model.

[0076] In this embodiment, the parameter loss function includes a parameter mean square error loss function, a parameter distance loss function, and a parameter mapping loss function generated based on the parameters at the corresponding positions in the training passing parameters and the labeled passing parameters.

[0077] Specifically, taking the passing parameters in this embodiment as including four parameters as an example, correspondingly, the parameter mean square error loss function includes a first parameter mean square error loss function formed based on the first training mean square error between the first two adjacent training passing parameters and the first labeled mean square error between the first two adjacent labeled passing parameters, and a second parameter mean square error loss function formed based on the second training mean square error between the last two adjacent training passing parameters and the second labeled mean square error between the last two adjacent labeled passing parameters.

[0078] Specifically, the parameter distance loss function includes a first parameter distance loss function based on the first two adjacent training passing parameters, and a second parameter distance loss function based on the last two adjacent training passing parameters.

[0079] Specifically, the parameter mapping loss function includes a parameter mapping loss function between the first mapping parameter obtained by mapping the training parameters to the α data field and the second mapping parameter obtained by mapping the labeled parameters to the α data field.

[0080] Based on the above embodiments, parameter weights corresponding to respective parameter loss functions are obtained. Among them, the weight values of the respective parameter weights are different. For example, the weights among the parameter mapping loss function, the parameter distance loss function, and the first parameter mean square error loss function are the same and higher than those of the second parameter mean square error loss function. The purpose is that in the actual application process, since the α data field is closer to the application scenario of volume rendering, the parameter mapping loss function can better reflect the semantic information of the image during volume rendering. Therefore, this loss function is relatively important and is set with a larger weight. Since increasing the spacing between parameter values will result in a blurred edge in the rendering result of volume rendering, the parameter distance loss function, as a forced loss function, needs to be set with a larger weight. Since the first two parameters are used to characterize the surface and edge features of volume rendering during image rendering, the first two parameters will affect the rendering result of volume rendering and need to be set with a larger weight. The last two parameters are used to characterize whether the inside of the blood vessel is solid or hollow during rendering. For this part, a certain degree of error of the model is allowed. Therefore, the weight of this part can be set smaller so that the model can pay more attention to the learning of the previous part and balance the learning ability of the model to obtain a better-performing transfer parameter determination model. Based on the above embodiments, since the first two parameters and the last two parameters in the transfer parameters represent different contents during volume rendering, different weights are also set for different data during the process of parameter mapping to the α data field. For example, the weights of the first two parameters are set greater than those of the last two parameters, which is also to improve the learning ability of the model for edge accuracy during rendering, so as to obtain a better-performing transfer parameter determination model.

[0081] It should also be noted that during the process of training the model, we introduced multiple loss functions into the head model and defined the relationship between the output parameters and the transfer function. Since the transfer function is a piecewise function and is not differentiable at some points, we gave an artificially differentiable definition through calculation, so that the model can avoid hard regression of data during the training process, and thus use the transfer function itself to convert the CT data field into the α data field according to the rendering channel information. The head model performs semantic extraction in two spaces of the CT data field and the α data field, which greatly avoids the overfitting speed of the model and enables the model to better understand the meaning of the α data field in the rendering pipeline. At the same time, since the four values defined by the transfer function need to have a strict partial order (size relationship), we introduced a partial order technique in the head output, used an activation function to transform the head output into four values with a strict partial order relationship, and associated these four values with the important parameters of the transfer function. Given that the model is accustomed to learning the numerical values themselves rather than the distances between the numerical values, and the control point distance affects the fogging effect near the rendered entity in the transfer function, we introduced a spacing loss to ensure that the model keeps the spacing as small as possible and reduces rendering fogging.

[0082] S130. Obtain the array parameter corresponding to the transfer parameter, and determine the transfer function based on the transfer parameter and the array parameter; the transfer function performs volume rendering on the original image.

[0083] In the embodiment of the present invention, the array parameter is one of the parameters constituting the transfer function, and is a value randomly generated within a preset range. Specifically, since the value range of the data channels in the α data field is 0-1, the array parameter is a value randomly generated between 0-1.

[0084] Specifically, the transfer parameter obtained based on the above embodiment and the randomly generated input array parameter are combined at corresponding positions to obtain the transfer function. Among them, the transfer function is used to perform volume rendering on the original image. Exemplarily, the expression form of the obtained transfer function can be an array (hu1, 0), (hu2, 0.98), (hu3, 1), (hu4, 0).

[0085] The technical solution of the embodiment of the present invention obtains the original image and the segmented image of the original image, and respectively determines the original cut image of the original image and the segmented cut image of the segmented image; determines the spliced image based on the original cut image and the segmented cut image, and inputs the spliced image into the pre-trained transfer parameter determination model to obtain the transfer parameter output by the model; obtains the array parameter corresponding to the transfer parameter, and determines the transfer function based on the transfer parameter and the array parameter; the transfer function performs volume rendering on the original image, solving the problem that the existing technology is time-consuming and has low accuracy in obtaining the transfer function by modulation method, realizing shortening the fine-tuning time of the transfer function and improving the accuracy of the transfer function.

[0086] Figure 2 It is a flowchart of another method for determining the transfer function of volume rendering provided by the embodiment of the present invention. Optionally, the technical solution provided in this embodiment can be combined with any of the above embodiments. On the basis of the above embodiments, the technical solution of this embodiment further includes: obtaining a pre-trained volume rendering model;

[0087] Input the original image and the transfer function into the volume rendering model to obtain the completed 3D effect image.

[0088] As Figure 2 shown, the method includes:

[0089] S210. Obtain the original image and the segmented image of the original image, and respectively determine the original cut image of the original image and the segmented cut image of the segmented image.

[0090] S220. Determine a stitched image based on the original cut image and the segmented cut image, and input the stitched image into a pre-trained transfer parameter determination model to obtain the transfer parameters output by the model.

[0091] S230. Obtain the array parameters corresponding to the transfer parameters, and determine a transfer function based on the transfer parameters and the array parameters; the transfer function performs volume rendering on the original image.

[0092] S240. Obtain a pre-trained volume rendering model, and input the original image and the transfer function into the volume rendering model to obtain a completed 3D effect image.

[0093] The technical solution of the embodiment of the present invention determines the transfer parameters of the transfer function through the transfer parameter determination model, generates the transfer function, realizes shortening the fine-tuning time of the transfer function, and then performs volume rendering on the original image core based on the transfer function to obtain a 3D effect image with more accurate rendering effect and improve the accuracy of image rendering.

[0094] Figure 3 It is a schematic structural diagram of a device for determining the transfer function of volume rendering provided by an embodiment of the present invention. As Figure 3 shown, the device includes: an image acquisition module 310, a transfer parameter determination module 320, and a transfer function determination module 330; wherein,

[0095] The image acquisition module 310 is configured to acquire the original image and the segmented image of the original image, and respectively determine the original cut image of the original image and the segmented cut image of the segmented image;

[0096] The transfer parameter determination module 320 is configured to determine a stitched image based on the original cut image and the segmented cut image, and input the stitched image into a pre-trained transfer parameter determination model to obtain the transfer parameters output by the model;

[0097] The transfer function determination module 330 is configured to obtain the array parameters corresponding to the transfer parameters, and determine a transfer function based on the transfer parameters and the array parameters; the transfer function performs volume rendering on the original image.

[0098] Optionally, on the basis of the above embodiments, the image acquisition module 310 includes:

[0099] A cut starting point determination unit, configured to determine a target object in the segmented image, and determine a cut starting point based on the target object;

[0100] The sliced image acquisition unit is configured to obtain a preset slicing size, and perform slicing processing on the original image and the segmented image respectively based on the slicing starting point and the slicing size, so as to obtain the original sliced image of the original image and the segmented sliced image of the segmented image.

[0101] Optionally, based on the above embodiments, the transfer parameter determination module 320 includes:

[0102] The segmented sliced image processing unit is configured to obtain the pixel values of each pixel point in the segmented sliced image, perform binarization processing on each pixel value based on at least one preset pixel threshold, and perform class marking on each pixel point in the segmented sliced image based on the binarization processing result, so as to obtain a segmented sliced processed image including at least one category;

[0103] The original sliced image processing unit is configured to perform image normalization processing on the original sliced image to obtain the original sliced processed image of the original sliced image;

[0104] The spliced image acquisition unit is configured to splice the segmented sliced processed image and the original sliced processed image to obtain the spliced image.

[0105] Optionally, based on the above embodiments, the spliced image acquisition unit includes:

[0106] The channel data determination subunit is configured to obtain the channel data of the original sliced processed image as the first channel data, and obtain the channel data of the segmented sliced processed image as the second channel data;

[0107] The spliced image determination subunit is configured to perform channel splicing on the first channel data and the second channel data to obtain a spliced image including two channels.

[0108] Optionally, based on the above embodiments, the device further includes: a model training module;

[0109] Specifically, the model training module includes:

[0110] The sample image and sample parameter acquisition unit is configured to obtain an annotated sample image and an annotated transfer parameter for training the transfer parameter determination model;

[0111] The training transfer parameter acquisition unit is configured to perform image processing on the annotated sample image to obtain a sample spliced image, and input the sample spliced image into the transfer parameter determination model to be trained, so as to obtain the training transfer parameter output by the model;

[0112] A model loss function determination unit, configured to determine at least one parameter loss function of the transfer parameter determination model in the current iteration based on the training transfer parameters and the annotation transfer parameters, and determine the model loss function of the transfer parameter determination model in the current iteration based on each of the parameter loss functions and the parameter weights respectively corresponding to the respective parameter loss functions; the weight values of the respective parameter weights are different;

[0113] A model training unit, configured to perform current model parameter adjustment on the transfer parameter determination model based on the model loss function, and continue iterative training based on the model after parameter adjustment until the iterative stop condition is met and then stop training to obtain a trained transfer parameter determination model.

[0114] Optionally, based on the above embodiments, the parameter loss function includes a parameter mean squared error loss function, a parameter distance loss function, and a parameter mapping loss function generated based on the parameters at corresponding positions in the training transfer parameters and the annotation transfer parameters.

[0115] Optionally, based on the above embodiments, the apparatus further includes:

[0116] A model acquisition module, configured to acquire a pre-trained volume rendering model;

[0117] An image drawing module, configured to input the original image and the transfer function into the volume rendering model to obtain a drawn 3D effect image.

[0118] The transfer function determination apparatus for volume rendering provided by the embodiments of the present invention can execute the transfer function determination method for volume rendering provided by any embodiment of the present invention, and has corresponding functional modules and beneficial effects for executing the method.

[0119] Figure 4 FIG. shows a schematic structural diagram of an electronic device 10 that can be used to implement the embodiments of the present invention. The electronic device is intended to represent various forms of digital computers, such as, a laptop computer, a desktop computer, a workbench, a personal digital assistant, a server, a blade server, a mainframe computer, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as, a personal digital processing, a cellular phone, a smart phone, a wearable device (such as a helmet, glasses, a watch, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are only examples and are not intended to limit the implementation of the present invention described and / or claimed herein.

[0120] As Figure 4As shown, the electronic device 10 includes at least one processor 11 and a memory communicatively connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc. The memory stores a computer program executable by the at least one processor. The processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 into the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0121] Multiple components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a magnetic disk, an optical disc, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0122] The processor 11 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the processor 11 include but are not limited to a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as the transfer function determination method for volume rendering.

[0123] In some embodiments, the transfer function determination method for volume rendering can be implemented as a computer program tangibly embodied in a computer-readable storage medium, such as the storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the transfer function determination method for volume rendering described above can be executed. Alternatively, in other embodiments, the processor 11 can be configured to execute the transfer function determination method for volume rendering in any other appropriate manner (e.g., by means of firmware).

[0124] The various embodiments of the systems and techniques described above in this specification can be implemented in digital electronic circuitry, integrated circuit systems, field programmable gate arrays (FPGA), application specific integrated circuits (ASIC), application specific standard products (ASSP), systems-on-chip (SOC), complex programmable logic devices (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include: being implemented in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which may be a special-purpose or general-purpose programmable processor that receives data and instructions from, and transmits data and instructions to, a storage system, at least one input device, and at least one output device.

[0125] The computer programs for implementing the methods of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the computer programs, when executed by the processor, cause the functions / operations specified in the flowchart and / or block diagram to be implemented. The computer programs may execute entirely on the machine, partly on the machine, as a stand-alone software package partly on the machine and partly on a remote machine or entirely on the remote machine or server.

[0126] In the context of the present invention, a computer-readable storage medium may be a tangible medium that can contain, or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. The computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, the computer-readable storage medium may be a machine-readable signal medium. More specific examples of the machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0127] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a trackball) through which the user can provide input to the electronic device. Other kinds of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0128] The systems and techniques described herein can be implemented in a computing system including backend components (e.g., as a data server), or a computing system including middleware components (e.g., an application server), or a computing system including frontend components (e.g., a user computer having a graphical user interface or a web browser through which the user can interact with an implementation of the systems and techniques described herein), or a computing system including any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected to each other by digital data communication in any form or medium (e.g., a communication network). Examples of communication networks include: local area network (LAN), wide area network (WAN), blockchain network, and the Internet.

[0129] The computing system can include a client and a server. The client and the server are generally far from each other and typically interact through a communication network. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system, solving the defects of difficult management and weak business scalability existing in traditional physical hosts and VPS services.

[0130] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps recited in the present invention can be executed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved, and no limitation is imposed herein.

[0131] The above specific embodiments do not constitute a limitation on the protection scope of the present invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A method for determining a transfer function for volume rendering, characterized in that Including: Obtain the original image and the segmented image of the original image, and respectively determine the original cut image of the original image and the segmented cut image of the segmented image; Determine a spliced image based on the original cut image and the segmented cut image, and input the spliced image into a pre-trained transfer parameter determination model to obtain the transfer parameters output by the model; Obtain the array parameters corresponding to the transfer parameters, and determine a transfer function based on the transfer parameters and the array parameters; the transfer function performs volume rendering on the original image.

2. The method according to claim 1, characterized in that, The step of respectively determining the original cut image of the original image and the segmented cut image of the segmented image includes: Determine the target object in the segmented image, and determine the cut starting point based on the target object; Obtain a preset cut size, and perform cut processing on the original image and the segmented image respectively based on the cut starting point and the cut size to obtain the original cut image of the original image and the segmented cut image of the segmented image.

3. The method according to claim 1, characterized in that The step of determining a spliced image based on the original cut image and the segmented cut image includes: Obtain the pixel values of each pixel point in the segmented cut image, perform binarization processing on each pixel value based on at least one preset pixel threshold, and perform category marking on each pixel point in the segmented cut image based on the binarization processing result to obtain a segmented cut processed image including at least one category; Perform image normalization processing on the original cut image to obtain the original cut processed image of the original cut image; Perform splicing processing on the segmented cut processed image and the original cut processed image to obtain the spliced image.

4. The method according to claim 3, characterized in that, The step of performing splicing processing on the segmented cut processed image and the original cut processed image to obtain the spliced image includes: Obtain the channel data of the original cut processed image as the first channel data, and obtain the channel data of the segmented cut processed image as the second channel data; Perform channel splicing on the first channel data and the second channel data to obtain a spliced image including two channels.

5. The method according to claim 1, wherein The training of the transfer parameter determination model includes: Obtain the labeled sample image and the labeled transfer parameters for training the transfer parameter determination model; Perform image processing on the labeled sample image to obtain a sample spliced image, and input the sample spliced image into the transfer parameter determination model to be trained to obtain the training transfer parameters output by the model; Determine at least one parameter loss function of the transfer parameter determination model in the current iteration based on the training transfer parameters and the labeled transfer parameters, and determine the model loss function of the transfer parameter determination model in the current iteration based on each parameter loss function and the parameter weights respectively corresponding to each parameter loss function; the weight values of each parameter weight are different; Perform current model parameter adjustment on the transfer parameter determination model based on the model loss function, and continue iterative training based on the model after parameter adjustment until the iterative stop condition is met and then stop training to obtain the trained transfer parameter determination model.

6. The method according to claim 5, characterized in that The parameter loss function includes a parameter mean square error loss function, a parameter distance loss function, and a parameter mapping loss function generated based on the parameters at corresponding positions in the training transfer parameters and the annotation transfer parameters.

7. The method according to claim 1, characterized in that, The method further includes: Obtaining a pre-trained volume rendering model; Inputting the original image and the transfer function into the volume rendering model to obtain a completed 3D effect image.

8. A transfer function determination device for volume rendering, characterized in that It includes: An image acquisition module for acquiring an original image and a segmented image of the original image, and respectively determining an original cut image of the original image and a segmented cut image of the segmented image; A transfer parameter determination module for determining a spliced image based on the original cut image and the segmented cut image, and inputting the spliced image into a pre-trained transfer parameter determination model to obtain transfer parameters output by the model; A transfer function determination module for obtaining an array parameter corresponding to the transfer parameter, and determining a transfer function based on the transfer parameter and the array parameter; the transfer function performs volume rendering on the original image.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the method for determining the transfer function of volume rendering according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing a processor to implement the method for determining the transfer function of volume rendering according to any one of claims 1-7 when executed.

Images