Focus ablation prediction method and device and computer readable storage medium

Abstract

The invention provides a focus ablation prediction method and device and a computer readable storage medium, and the method comprises the steps: obtaining a pre-operation image and an operation plan image, the pre-operation image comprises an operation focus part, and the operation plan image comprises an operation region and a plurality of needle holes; preprocessing the surgical scheme image to obtain influence areas of a plurality of pinholes; and inputting the preoperative image and the preprocessed operation plan image into a pre-trained prediction model to obtain a focus ablation prediction result. The invention provides a technical scheme for predicting a postoperative focus ablation result before an operation so as to assist in optimizing an operation scheme.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a lesion ablation prediction method and device, and a computer-readable storage medium. Background Art

[0002] In the field of cancer treatment, one of the current cutting-edge clinical treatment technologies is High Frequency Irreversible Electroporation (H-FIRE). The treatment method involves puncturing multiple needle electrodes in the treatment area and using electric pulses to ablate the lesions. The ablation results need to be obtained through magnetic resonance imaging (MRI) imaging after one or several weeks. As an emerging application, it is currently in the early and exploratory stages of clinical application.

[0003] However, predicting the postoperative lesion ablation results before surgery to assist in optimizing the surgical plan is a technical problem that needs to be solved urgently. Summary of the invention

[0004] The present application provides a technical solution for predicting the postoperative lesion ablation results before surgery to assist in optimizing the surgical plan.

[0005] In order to achieve the above objectives, this application provides the following technical solutions:

[0006] In a first aspect, a lesion ablation prediction method is provided, the lesion ablation prediction method comprising: acquiring a preoperative image and a surgical plan image, the preoperative image comprising a surgical lesion site, the surgical plan image comprising a surgical area and a plurality of pinholes; preprocessing the surgical plan image to obtain an impact area of ​​the plurality of pinholes; inputting the preoperative image and the preprocessed surgical plan image into a pre-trained prediction model to obtain a lesion ablation prediction result.

[0007] Optionally, the preprocessing of the surgical plan image includes: for each pixel in the surgical plan image that is within the influence area of ​​each pinhole, determining a pixel value of the pixel according to a pixel distance between the pixel and each pinhole.

[0008] Optionally, the prediction model includes an encoder, the encoder includes a first branch network and a second branch network, the first branch network is used to extract first low-level features and first high-level features of the preoperative image, and the second branch network is used to extract second low-level features and second high-level features of the surgical plan image.

[0009] Optionally, the prediction model also includes a decoder, the decoder including a first prediction head, a second prediction head and a third prediction head, the first prediction head is used to determine a first prediction image based on the first low-level features and the first high-level features, the second prediction head is used to determine a second prediction image based on the second low-level features and the second high-level features, the third prediction head is used to determine a fusion image based on the first low-level features, the first high-level features, the second low-level features and the second high-level features, and the lesion ablation prediction result includes the fusion image.

[0010] Optionally, the third prediction head calculates a first sum and a first product of the first low-level feature and the second low-level feature, calculates a second sum and a second product of the first high-level feature and the second high-level feature, and performs a convolution operation on the first sum, the first product, the second sum and the second product to obtain the fused image.

[0011] Optionally, in the process of training the prediction model using training data, the total loss of the prediction model is a weighted sum of a first cross entropy loss between the first predicted image and an actual predicted image in the training data, a second cross entropy loss between the second predicted image and the actual predicted image, and a third cross entropy loss between the fused image and the actual predicted image.

[0012] Optionally, before inputting the preoperative image and the preprocessed surgical plan image into the pre-trained prediction model, the method also includes: calculating the minimum rectangle covering the surgical lesion site based on the marked area for the surgical lesion site in the preoperative image; enlarging the minimum rectangle according to a preset ratio, and using the enlarged minimum rectangle to capture the surgical plan image.

[0013] In a second aspect, the present application also discloses a lesion ablation prediction device, which includes: an acquisition module, which acquires preoperative images and surgical plan images, wherein the preoperative images include the surgical lesion site, and the surgical plan images include the surgical area and multiple pinholes; a preprocessing module, which is used to preprocess the surgical plan images to obtain the influence areas of the multiple pinholes; and a prediction module, which is used to input the preoperative images and the preprocessed surgical plan images into a pre-trained prediction model to obtain a lesion ablation prediction result.

[0014] Optionally, the preprocessing module is used to determine the pixel value of each pixel located within the influence area of ​​each pinhole in the surgical plan image according to the pixel distance between the pixel and each pinhole.

[0015] Optionally, the prediction model includes an encoder, the encoder includes a first branch network and a second branch network, the first branch network is used to extract first low-level features and first high-level features of the preoperative image, and the second branch network is used to extract second low-level features and second high-level features of the surgical plan image.

[0016] Optionally, the prediction model also includes a decoder, the decoder including a first prediction head, a second prediction head and a third prediction head, the first prediction head is used to determine a first prediction image based on the first low-level features and the first high-level features, the second prediction head is used to determine a second prediction image based on the second low-level features and the second high-level features, the third prediction head is used to determine a fusion image based on the first low-level features, the first high-level features, the second low-level features and the second high-level features, and the lesion ablation prediction result includes the fusion image.

[0017] Optionally, the third prediction head calculates a first sum and a first product of the first low-level feature and the second low-level feature, calculates a second sum and a second product of the first high-level feature and the second high-level feature, and performs a convolution operation on the first sum, the first product, the second sum and the second product to obtain the fused image.

[0018] Optionally, in the process of training the prediction model using training data, the total loss of the prediction model is a weighted sum of a first cross entropy loss between the first predicted image and an actual predicted image in the training data, a second cross entropy loss between the second predicted image and the actual predicted image, and a third cross entropy loss between the fused image and the actual predicted image.

[0019] Optionally, the preprocessing module is also used to calculate the minimum rectangle covering the surgical lesion site based on the marked area for the surgical lesion site in the preoperative image; expand the minimum rectangle according to a preset ratio, and use the expanded minimum rectangle to capture the surgical plan image.

[0020] According to a third aspect, a computer-readable storage medium is provided, on which a computer program is stored. The computer program is executed by a processor to execute any one of the methods provided in the first aspect or the second aspect.

[0021] In a fourth aspect, a communication device is provided, comprising a memory and a processor, wherein the memory stores a computer program executable on the processor, and the processor executes the computer program to execute any one of the methods provided in the first aspect.

[0022] According to a fifth aspect, a computer program product is provided, on which a computer program is stored, and the computer program is executed by a processor to execute any one of the methods provided in the first aspect or the second aspect.

[0023] In a sixth aspect, an embodiment of the present application further provides a chip (or a data transmission device) on which a computer program is stored, and when the computer program is executed by the chip, the steps of the above method are implemented.

[0024] In the seventh aspect, an embodiment of the present application also provides a system chip, which is applied to a terminal, and the chip system includes at least one processor and an interface circuit, the interface circuit and the at least one processor are interconnected through lines, and the at least one processor is used to execute instructions to execute any one of the methods provided in the first aspect or the second aspect.

[0025] Compared with the prior art, the technical solution of this application has the following beneficial effects:

[0026] In the technical solution of this application, the preoperative image can reflect the condition of the lesion, and the surgical plan image can reflect the treatment plan for the lesion. By using the preoperative image and the surgical plan image, the postoperative lesion ablation result can be predicted to optimize the surgical plan and meet the needs of preoperative efficacy prediction. At the same time, it provides a reference solution for the prediction of postoperative lesion ablation in cancer, such as prostate cancer treatment, using high-frequency irreversible electroporation technology. In addition, by preprocessing the surgical plan image to obtain the influence area of ​​multiple pinholes, the range of the pinholes is reasonably expanded, so that the prediction model can more accurately identify the pinholes in the treatment plan, further ensuring the accuracy of the prediction of postoperative lesion ablation results.

[0027] Furthermore, in the technical solution of the present application, the third prediction head determines the fused image according to the first low-level feature, the first high-level feature, the second low-level feature and the second high-level feature, and the lesion ablation prediction result includes the fused image. The prediction model constructed by the technical solution of the present application can better extract the features in the preoperative image and the surgical plan image, unify the statistical characteristics of the images of the two, thereby achieving better feature fusion, and further ensuring the accuracy of the prediction of the postoperative lesion ablation result. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 is a flow chart of a lesion ablation prediction method provided in an embodiment of the present application;

[0029] Figure 2 This is a flow chart of a model training provided in an embodiment of the present application;

[0030] Figure 3 is a schematic diagram of a surgical plan image provided in an embodiment of the present application;

[0031] Figure 4 It is a structural schematic diagram of a prediction model provided in an embodiment of the present application;

[0032] Figure 5 This is a specific flow chart of a lesion ablation prediction method provided in an embodiment of the present application.

[0033] Figure 6 It is a structural schematic diagram of a lesion ablation prediction device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0034] As described in the background technology, predicting the postoperative lesion ablation results before surgery to assist in optimizing the surgical plan is a technical problem that needs to be solved urgently.

[0035] In the prior art, research on the evaluation of the efficacy of tumor surgery has mostly focused on the classification of postoperative indicators and negative and positive lesions.

[0036] This application aims at the scenario of lesion ablation prediction, and realizes the segmentation of surgical lesion sites, such as lesion ablation results in the prostate region, by fusing dual-modality images with a deep learning model. The applicant found that the pinholes in the surgical plan images manually annotated by doctors are extremely small and difficult for the model to recognize.

[0037] In view of the above difficulties, this application pre-processes the surgical plan image to reasonably expand the pinhole influence range.

[0038] The applicant also found that the statistical characteristics of preoperative images and surgical plan images are very different, making them difficult to directly fuse.

[0039] To address the above difficulties, this application introduces Atrous Spatial Pyramid Pooling (ASPP) based on a deep residual network, which can expand the receptive field and improve segmentation accuracy without losing resolution. At the same time, it introduces feature grouping and three-way supervision to construct a prediction model to help the model better extract and fuse image features.

[0040] This application predicts the range of lesion ablation to assist in planning a more reasonable electrode layout for surgical procedures, thereby achieving a more ideal ablation area morphology. This application uses a deep learning method to predict the postoperative lesion ablation results using the patient's preoperative images and surgical plan to optimize the surgical plan and meet the needs of preoperative efficacy prediction.

[0041] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the specific embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0042] See also Figure 1 The method provided in this application specifically includes the following steps:

[0043] Step 101: Obtain a preoperative image and a surgical plan image, wherein the preoperative image includes the surgical lesion site, and the surgical plan image includes the surgical area and a plurality of pinholes.

[0044] Specifically, the preoperative image may be a magnetic resonance imaging (MRI). Generally speaking, the preoperative image includes information of the entire organ of the patient.

[0045] Taking prostate cancer lesion ablation surgery as an example, a magnetic resonance imaging device is used to collect the patient's prostate MRI image before the operation, that is, the MRI image includes information about the prostate.

[0046] Specifically, the surgical plan image may be an image obtained by pre-marking the surgical lesion site by medical workers, and the pinholes in the surgical plan image indicate the positions where the electrodes need to be inserted. The surgical plan image may reflect the surgical plan for the lesion site.

[0047] Step 102: Preprocess the surgical plan image to obtain the influence areas of multiple pinholes.

[0048] The embodiment of the present application can reasonably expand the pinhole influence range by preprocessing the surgical plan image, which helps the prediction model to identify the pinhole in the surgical plan image, thereby helping to improve the prediction effect.

[0049] Step 103: input the preoperative image and the preprocessed surgical plan image into the pre-trained prediction model to obtain the lesion ablation prediction result.

[0050] In a specific implementation, the prediction model can be pre-trained using training data, and the training data includes preoperative images, surgical plan images, and postoperative review images. Among them, the postoperative review image is an image of the lesion ablation result of the patient after a period of time (for example, one week or several weeks) after surgery, and the postoperative review image includes the surgical lesion site. The postoperative review image can reflect the actual result of lesion ablation.

[0051] The embodiments of the present application can predict the results of postoperative lesion ablation by using preoperative images and surgical plan images, so as to optimize the surgical plan and meet the needs of preoperative efficacy prediction. At the same time, it provides a reference solution for the prediction of postoperative lesion ablation using high-frequency irreversible electroporation technology in cancer, such as prostate cancer treatment. In addition, by preprocessing the surgical plan images to obtain the influence areas of multiple pinholes, the range of the pinholes is reasonably expanded, so that the prediction model can more accurately identify the pinholes in the treatment plan, further ensuring the accuracy of the prediction of postoperative lesion ablation results.

[0052] It should be pointed out that the serial numbers of the steps in this embodiment do not limit the execution order of the steps.

[0053] It is understandable that, in a specific implementation, the lesion ablation prediction method can be implemented in the form of a software program, and the software program runs in a processor integrated inside a chip or a chip module. The method can also be implemented in the form of software combined with hardware, which is not limited in this application.

[0054] The construction and training process of the prediction model is described below in conjunction with specific embodiments.

[0055] Please refer to Figure 2 , Figure 2 A model training method is shown.

[0056] In step 201, training data is obtained. The training data includes preoperative images, surgical plan images, and postoperative review images.

[0057] In step 202, the surgical plan image is preprocessed to obtain the influence areas of multiple pinholes.

[0058] In a specific implementation, for each pixel in the surgical plan image within the influence area of ​​each pinhole, the pixel value of the pixel is determined according to the pixel distance between the pixel and each pinhole, wherein the smaller the sum of the pixel distances between the pixel and each pinhole, the larger the pixel value of the pixel.

[0059] Specifically, the influence area range of each pinhole can be represented by the influence area radius, and the center of the influence area of ​​each pinhole is the pinhole. The influence area radius can be preset.

[0060] Specifically, the pinhole is expanded according to the pinhole position marked in the surgical plan image. The surgical plan image is converted into a grayscale image. Figure 3 As shown in Figure a, white pixels represent pinholes and the box shows the magnified area. Get the coordinates of all white pixels.

[0061] For each white pixel, the pixel value calculation formula is shown in formula (1):

[0062]

[0063] Where F(x,y) represents the pixel value of the pixel point (x,y), i is the number of the pinhole, n is the total number of pinholes, and D i (x,y) represents the straight-line distance between the pixel point (x,y) and each pinhole, trans represents the transparency percentage, and radius represents the radius of the affected area.

[0064] Preferably, when the transparency percentage is 50% and the impact area radius is 10 pixels, the prediction effect of the prediction model is better.

[0065] Furthermore, using the surgical plan image as a mask can prevent the affected area of ​​each pinhole from exceeding the marked range.

[0066] The processed surgical plan image is as follows Figure 3 As shown in Figure b, it can be seen from the figure that compared with the surgical plan image before processing, the affected area of ​​each pinhole is larger.

[0067] In step 203, the preoperative image is preprocessed to capture the surgical lesion site.

[0068] In a specific implementation, since the range of MRI images is usually relatively large, in order to effectively predict the effect of lesion ablation, the MRI images may be intercepted.

[0069] Specifically, taking the prostate as an example of a surgical lesion, the smallest rectangle that can enclose the area is calculated based on the marked prostate area, and the rectangle is expanded along the long side as the interception range, and expanded outward from the center to a preset ratio, such as 144%, to retain more relevant information in the MRI.

[0070] Furthermore, the cropped MRI image is resampled to a fixed size, such as 330 pixels×330 pixels.

[0071] It should be noted that the preset ratio, resampling size, etc. can be adaptively set according to the actual application scenario, and this application does not impose any restrictions on this.

[0072] Furthermore, data enhancement can be performed on the preoperative images and surgical plan images to expand the training data. Specifically, the preoperative images and surgical plan images can be rotated, flipped upside down, flipped left to right, etc., so that the number and type of images in the training data are expanded.

[0073] In step 204, a prediction model is constructed, wherein the prediction model may be a bimodal "encoder-decoder" structure.

[0074] In this embodiment, the encoder includes a first branch network and a second branch network, the first branch network is used to extract first low-level features and first high-level features of the preoperative image, and the second branch network is used to extract second low-level features and second high-level features of the surgical plan image.

[0075] Please refer to Figure 4The encoder part includes two branch networks with the same structure (i.e., the first branch network and the second branch network), which serve as the preoperative image input and surgical plan image input branches respectively. Each branch uses a residual neural network, such as ResNet18, as the backbone network. In order to capture multi-scale information, a feature pyramid (atrous spatial pyramid pooling, ASPP) is added at the end. The feature output of the residual neural network hierarchy is divided into high-level features and low-level features, where the outputs of Layer1 and Layer2 are used as low-level features, and the outputs of Layer3 and the ASPP at the end are used as high-level features to improve the compatibility of features.

[0076] Specifically, in the residual neural network, convolution, batch normalization, and activation function ReLU (referred to as Bconv operation) are first converted into a unified channel feature, and then the group is merged to save memory and computational costs to obtain the first low-level feature of the preoperative image after the merged output result. First high-level features of preoperative images Second-level features of surgical plan images Second-level features of surgical planning images Due to the symmetry of the structure, the formulas for the previous image branch are shown in formulas (2) and (3):

[0077]

[0078] Among them, Up represents bilinear upsampling, represents the first low-level features of the preoperative image output by layer 1, represents the second low-level features of the preoperative image output by layer 2, represents the first high-level features of the preoperative image output by layer 3, Represents the second high-level features of the preoperative image output by ASPP.

[0079] The low-level features referred to in this embodiment are basic information directly extracted from the original data of the image, which are usually related to the physical properties of the image and include the basic structure and texture information of the image.

[0080] The high-level features referred to in this embodiment are abstract information further extracted from low-level features, and are usually related to the semantic content of the image.

[0081] It should be noted that the specific number of layers of the residual neural network can be set according to the actual application scenario, and this application does not impose any restrictions on this.

[0082] In this embodiment, the prediction model further includes a decoder, the decoder includes a first prediction head, a second prediction head and a third prediction head, the first prediction head is used to predict the first low-level feature and the first advanced feature Determine the first predicted image S m The second prediction head is used to predict the second low-level features and the second advanced feature Determine the second predicted image S p The third prediction head is used to determine the fused image S according to the first low-level feature, the first high-level feature, the second low-level feature and the second high-level feature. c .

[0083] Continue to refer to Figure 4 The decoder part horizontally connects the input low-level features and high-level features and passes them through the prediction head. Each prediction head includes two convolution (Bconv) operations and a (1×1,1) convolution, an activation function (sigmoid) layer and an upsampling operation to obtain the first branch network and the second branch network. The two branches have their own first prediction image S m and the second predicted image S p .

[0084] In order to achieve bimodal feature fusion, the third prediction head adds and multiplies the low-level features and high-level features of the two branches hierarchically, connects the results horizontally and then passes through the Bconv operation. The two results are connected and then passed through the prediction head to obtain the fused image S c , as shown in formulas (4)-(6),

[0085]

[0086]

[0087] Among them, P represents the series of operations performed by the prediction head, and Up represents bilinear upsampling.

[0088] The first predicted image S is implemented using the standard cross entropy loss function m , the second predicted image S p and the fused image S c The three-way supervision is as shown in formula (7):

[0089]

[0090] in, is the total loss, is the cross entropy loss, G represents the postoperative review image, λ x is the weight value for each output image branch, S c The final image prediction result.

[0091] Preferably, the weight λ m and λ p Set to 0.5, λ c Set to 1 to perform backpropagation of loss.

[0092] The algorithm proposed in the present invention preprocesses the data set, uses a dual-modal input deep residual network to perform low-level and high-level feature extraction respectively, utilizes grouping features to obtain three predicted images and supervises the output branches respectively, which can improve the segmentation accuracy of the prediction model.

[0093] In step 205, the prediction model is trained and verified.

[0094] In a specific embodiment, the actual data set collected is 71 groups, 213 images. The data enhancement stage performs upside down and left-right flipping and 9 36° rotations on each group of images, amplifying them 12 times in total. The final experimental data set is 852 groups, 2556 images. During the experiment, the training set and the test set are divided into 9:1 ratios, and a ten-fold cross validation is performed.

[0095] Furthermore, the obtained prediction models can be evaluated and compared using common image segmentation indicators such as accuracy, precision, recall, F1 index, F2 index, average intersection-over-union ratio, and common medical evaluation indicators. At the same time, the model is cross-validated by 10-fold to calculate the average performance index of the model and evaluate the generalization ability of the model. The higher the value of the above indicators, the better the model effect. The specific verification results are shown in Table 1.

[0096] Table 1

[0097]

[0098]

[0099] As shown in Table 1, by comparing the salient target detection model BTSNet with a depth of 50 layers, the salient target detection model BTSNet with a depth of 18 layers, the convolutional neural network ResNet18 with a depth of 18 layers, and the prediction model of the present application in terms of various indicators, it can be seen that the prediction model of the present application has better performance in terms of accuracy, precision, recall rate, F1 index, F2 index, average intersection-over-union ratio and other indicators.

[0100] The above embodiment shows the training and verification process of the prediction model. After the training and verification are completed, the prediction model can predict the input preoperative images and surgical plan images and output the lesion ablation prediction results.

[0101] Please refer to Figure 5In step 501, an input image is acquired, and the input image includes a preoperative image and a surgical plan image.

[0102] In step 502, the surgical plan image is preprocessed to obtain the influence areas of multiple pinholes.

[0103] In step 503, the preoperative image is preprocessed to capture the surgical lesion site.

[0104] In step 504, the preoperative image and the preprocessed surgical plan image are input into a pre-trained prediction model to obtain a lesion ablation prediction result.

[0105] In this embodiment, the prediction model includes an encoder and a decoder, the encoder includes a first branch network and a second branch network, and the decoder includes a first prediction head, a second prediction head, and a third prediction head.

[0106] In this embodiment, the first branch network is used to extract the first low-level features of the preoperative image and the first advanced feature The second branch network is used to extract the second low-level features of the surgical plan image and the second advanced feature

[0107] In this embodiment, the first prediction head is used to predict the first low-level feature and the first advanced feature Determine the first predicted image S m The second prediction head is used to predict the second low-level features and the second advanced feature Determine the second predicted image S p The third prediction head is used to determine the fused image S according to the first low-level feature, the first high-level feature, the second low-level feature and the second high-level feature. c , the lesion ablation prediction results include the fusion image S c That is, the fused image S c is the final image prediction result of the prediction model for the input image, and the fused image S c It can reflect the result of lesion ablation after the surgical plan shown in the surgical plan image is adopted for the preoperative image.

[0108] The algorithm proposed in the present invention preprocesses the data set, uses a dual-modal input deep residual network to perform low-level and high-level feature extraction respectively, utilizes grouping features to obtain three predicted images and supervises the output branches respectively, which can improve the segmentation accuracy of the prediction model.

[0109] Those skilled in the art will appreciate that steps S501 to S503 may be considered as similar to the above Figure 3The steps S301 to S303 in the embodiment shown are corresponding execution steps, and the two complement each other in specific implementation principles and logic. Figure 3 The relevant description of the illustrated embodiment will not be repeated here.

[0110] Please refer to Figure 6 , Figure 6 A lesion ablation prediction device 60 is shown, and the lesion ablation prediction device 60 may include:

[0111] An acquisition module 601 acquires a preoperative image and a surgical plan image, wherein the preoperative image includes the surgical lesion site, and the surgical plan image includes the surgical area and a plurality of pinholes;

[0112] A preprocessing module 602 is used to preprocess the surgical plan image to obtain the influence areas of multiple pinholes;

[0113] The prediction module 603 is used to input the pre-operative images and the pre-processed surgical plan images into the pre-trained prediction model to obtain the lesion ablation prediction results.

[0114] This embodiment can predict the postoperative lesion ablation results by using preoperative images and surgical plan images to optimize the surgical plan and meet the needs of preoperative efficacy prediction. At the same time, it provides a reference solution for the prediction of postoperative lesion ablation using high-frequency irreversible electroporation technology in cancer, such as prostate cancer treatment. In addition, by preprocessing the surgical plan images to obtain the affected areas of multiple pinholes, the range of the pinholes is reasonably expanded, so that the prediction model can more accurately identify the pinholes in the treatment plan, further ensuring the accuracy of the prediction of postoperative lesion ablation results.

[0115] Furthermore, the preprocessing module 602 is used to determine the pixel value of each pixel located within the influence area of ​​each pinhole in the surgical plan image according to the pixel distance between the pixel and each pinhole.

[0116] Furthermore, the preprocessing module 602 is used to calculate the minimum rectangle covering the surgical lesion site according to the marked area for the surgical lesion site in the preoperative image; expand the minimum rectangle according to a preset ratio, and use the expanded minimum rectangle to intercept the surgical plan image.

[0117] Furthermore, the prediction model includes an encoder, which includes a first branch network and a second branch network, the first branch network is used to extract first low-level features and first high-level features of the preoperative image, and the second branch network is used to extract second low-level features and second high-level features of the surgical plan image.

[0118] Furthermore, the prediction model also includes a decoder, which includes a first prediction head, a second prediction head and a third prediction head. The first prediction head is used to determine a first prediction image based on the first low-level feature and the first high-level feature. The second prediction head is used to determine a second prediction image based on the second low-level feature and the second high-level feature. The third prediction head is used to determine a fusion image based on the first low-level feature, the first high-level feature, the second low-level feature and the second high-level feature. The lesion ablation prediction result includes a fusion image.

[0119] In a specific implementation, the above-mentioned lesion ablation prediction device 60 can correspond to a chip with a lesion ablation prediction function in a terminal device, such as a system-on-a-chip (SOC), a baseband chip, etc.; or correspond to a chip module with a lesion ablation prediction function in the terminal device; or correspond to a chip module with a data processing function chip, or correspond to a terminal device.

[0120] For other relevant descriptions about the lesion ablation prediction device 60 , reference may be made to the relevant descriptions in the aforementioned embodiments, which will not be repeated here.

[0121] Regarding the various modules / units included in the various devices and products described in the above embodiments, they can be software modules / units, or hardware modules / units, or they can be partially software modules / units and partially hardware modules / units. For example, for various devices and products applied to or integrated in a chip, the various modules / units included therein can all be implemented in the form of hardware such as circuits, or at least some of the modules / units can be implemented in the form of software programs, which run on a processor integrated inside the chip, and the remaining (if any) modules / units can be implemented in the form of hardware such as circuits; for various devices and products applied to or integrated in a chip module, the various modules / units included therein can all be implemented in the form of hardware such as circuits, and different modules / units can be located in the same component of the chip module (such as a chip, circuit module, etc.) or in different components, or at least some of the modules / units can be implemented in the form of software programs. It is implemented in the form of a software program, which runs on a processor integrated inside the chip module, and the remaining (if any) modules / units can be implemented in hardware such as circuits; for various devices and products applied to or integrated in the terminal equipment, the various modules / units contained therein can be implemented in hardware such as circuits, and different modules / units can be located in the same component (for example, chip, circuit module, etc.) or in different components in the terminal equipment, or, at least some modules / units can be implemented in the form of a software program, which runs on a processor integrated inside the terminal equipment, and the remaining (if any) modules / units can be implemented in hardware such as circuits.

[0122] The embodiment of the present application also discloses a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is run, the steps of the method shown in the above embodiment can be executed. The storage medium may include a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc. The storage medium may also include a non-volatile memory (non-volatile) or a non-transitory memory, etc.

[0123] It should be understood that the term "and / or" in this article is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article indicates that the associated objects before and after are in an "or" relationship.

[0124] The "plurality" appearing in the embodiments of the present application refers to two or more.

[0125] The first, second, etc. descriptions appearing in the embodiments of the present application are only used for illustration and distinction of the description objects. There is no order, nor do they indicate any special limitation on the number of devices in the embodiments of the present application, and cannot constitute any limitation on the embodiments of the present application.

[0126] The "connection" that appears in the embodiments of the present application refers to various connection methods such as direct connection or indirect connection to achieve communication between devices, and the embodiments of the present application do not impose any limitations on this.

[0127] The above embodiments may be implemented in whole or in part by software, hardware, firmware or any other combination thereof. When implemented using software, the above embodiments may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired or wireless means.

[0128] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0129] In the several embodiments provided in the present application, it should be understood that the disclosed methods, devices and systems can be implemented in other ways. For example, the device embodiments described above are merely schematic; for example, the division of the units is only a logical function division, and there may be other division methods in actual implementation; for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0130] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0131] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may be physically included separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0132] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform some steps of the method described in each embodiment of the present application.

[0133] Although the present application is disclosed as above, the present application is not limited thereto. Any person skilled in the art can make various changes and modifications without departing from the spirit and scope of the present application. Therefore, the protection scope of the present application shall be subject to the scope defined by the claims.

Claims

1. A lesion ablation prediction method, characterized in that: include: Acquire a preoperative image and a surgical plan image, wherein the preoperative image includes a surgical lesion site, and the surgical plan image includes a surgical area and a plurality of pinholes; Preprocessing the surgical plan image to obtain the influence areas of the multiple pinholes; The preoperative images and the preprocessed surgical plan images are input into a pre-trained prediction model to obtain lesion ablation prediction results.

2. The method for predicting lesion ablation according to claim 1, characterized in that: The preprocessing of the surgical plan image comprises: For each pixel in the surgical plan image that is within the influence area of ​​each pinhole, a pixel value of the pixel is determined according to a pixel distance between the pixel and each pinhole.

3. The method for predicting lesion ablation according to claim 1, characterized in that: The prediction model includes an encoder, which includes a first branch network and a second branch network, the first branch network is used to extract first low-level features and first high-level features of the preoperative image, and the second branch network is used to extract second low-level features and second high-level features of the surgical plan image.

4. The method for predicting lesion ablation according to claim 3, characterized in that: The prediction model also includes a decoder, which includes a first prediction head, a second prediction head and a third prediction head. The first prediction head is used to determine a first prediction image based on the first low-level features and the first high-level features. The second prediction head is used to determine a second prediction image based on the second low-level features and the second high-level features. The third prediction head is used to determine a fused image based on the first low-level features, the first high-level features, the second low-level features and the second high-level features. The lesion ablation prediction result includes the fused image.

5. The method for predicting lesion ablation according to claim 4, characterized in that: The third prediction head calculates a first sum and a first product of the first low-level feature and the second low-level feature, calculates a second sum and a second product of the first high-level feature and the second high-level feature, and performs a convolution operation on the first sum, the first product, the second sum, and the second product to obtain the fused image.

6. The method for predicting lesion ablation according to claim 4, characterized in that: In the process of training the prediction model using training data, the total loss of the prediction model is a weighted sum of a first cross entropy loss between the first predicted image and an actual predicted image in the training data, a second cross entropy loss between the second predicted image and the actual predicted image, and a third cross entropy loss between the fused image and the actual predicted image.

7. The method for predicting lesion ablation according to claim 1, characterized in that: Before inputting the preoperative image and the preprocessed surgical plan image into the pre-trained prediction model, the method further includes: calculating the minimum rectangle covering the surgical lesion site according to the annotated area for the surgical lesion site in the preoperative image; The minimum rectangle is enlarged according to a preset ratio, and the surgical plan image is captured using the enlarged minimum rectangle.

8. A lesion ablation prediction device, characterized in that: include: An acquisition module, which acquires a preoperative image and a surgical plan image, wherein the preoperative image includes a surgical lesion site, and the surgical plan image includes a surgical area and a plurality of pinholes; A preprocessing module, used for preprocessing the surgical plan image to obtain the influence areas of the multiple pinholes; The prediction module is used to input the preoperative image and the preprocessed surgical plan image into a pre-trained prediction model to obtain a lesion ablation prediction result.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the lesion ablation prediction method according to any one of claims 1 to 7 are executed.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the lesion ablation prediction method according to claims 1 to 7 are implemented.

11. A terminal device, comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, characterized in that: When the processor runs the computer program, the steps of the lesion ablation prediction method according to any one of claims 1 to 7 are performed.

Images