Method, apparatus, device, and medium for associating imaging images and pathological images

Abstract

The present invention discloses a method for associating imaging images with pathological images, including: first, obtaining a first true imaging image of a diseased tissue, a second true imaging image of a diseased tissue sample, and a true pathological image of the diseased tissue sample. Then, fine-tuning the converged generative adversarial network according to the second true imaging image and the true pathological image until the generative adversarial network generates a virtual pathological image based on the true imaging image of the target object, and the virtual pathological image is similar to the true pathological image of the target object. Finally, generating a virtual pathological image of the diseased tissue from the first true imaging image through the fine-tuned generative adversarial network, so that for the diseased tissue, the imaging image and the pathological image can be closely associated at the image level, promoting the associated research between macro and micro. In addition, an apparatus, a device, and a storage medium for associating imaging images with pathological images are also proposed.

Description

Technical Field

[0001] The present invention relates to the technical field of medical images, and in particular to a method, device, equipment and medium for associating imaging images with pathological images. Background Art

[0002] Pathology is a medical practice science for pathologically diagnosing diseased tissue samples and cells taken from a patient's living body, including steps such as isolating biological tissues, chemical and immunological staining, and observing under a microscope or a digital pathology scanner. Correspondingly, pathological images are fine images at the microscopic cell level and tissue level.

[0003] Medical Imaging, on the other hand, studies the interaction between a certain medium (such as X-rays, electromagnetic fields, ultrasonic waves, etc.) and the human body to present the internal tissue and organ structures and densities of the human body in the form of images. Compared with pathological images, imaging images are more macroscopic and holistic.

[0004] In the same case, these two types of images often appear in the doctor's field of vision from different dimensions and imaging methods. For a certain diseased tissue, the association between traditional pathological images and imaging images is mostly a simple text-based association, lacking a close association at the image level. This also leads to insufficient connection between pathological images and imaging images, restricting the macroscopic and microscopic association research between medical images. Summary of the Invention

[0005] Based on this, it is necessary to provide a method, device, equipment and medium for associating imaging images with pathological images to solve the problem of insufficient connection between pathological images and imaging images.

[0006] A method for associating imaging images with pathological images, the method comprising:

[0007] Obtain a first true imaging image of a diseased tissue of a target object, a second true imaging image of the diseased tissue sample, and a true pathological image of the diseased tissue sample; wherein, the diseased tissue sample is obtained by performing core needle biopsy on the diseased tissue;

[0008] Fine-tune a converged generative adversarial network according to the second true imaging image and the true pathological image; wherein, the fine-tuned generative adversarial network generates a virtual pathological image based on the true imaging image of the target object, and the virtual pathological image is similar to the true pathological image of the target object;

[0009] The first real imaging image is generated into a virtual pathological image of the diseased tissue by a fine-tuned generative adversarial network.

[0010] In one specific embodiment, the generative adversarial network includes a generator and a discriminator. The method further includes:

[0011] Obtaining multiple sets of training image sets based on multiple training objects; wherein, one set of training image sets consists of a real imaging training image and a real pathological training image, and the real imaging training image and the real pathological training image in the same set of training image sets correspond to the same position in the diseased tissue sample of the same training object;

[0012] Inputting a real imaging training image into the generator to obtain a generated virtual pathological training image;

[0013] Inputting the virtual pathological training image and the real pathological training image in the same set of training image sets into the discriminator to obtain a first discrimination result, and iteratively adjusting the parameters of the generator and the discriminator according to the first discrimination result to increase the probability of the discriminator making a correct discrimination and increase the probability of the image generated by the generator making the discriminator make a wrong discrimination. Then return to execute the step of inputting a real imaging training image into the generator to obtain a generated virtual pathological training image until the generative adversarial network converges; wherein, the first discrimination result is used to indicate whether the virtual pathological training image and the real pathological training image are generated images.

[0014] In one specific embodiment, the step of inputting the virtual pathological training image and the real pathological training image in the same set of training image sets into the discriminator to obtain a first discrimination result, and iteratively adjusting the parameters of the generator and the discriminator according to the first discrimination result includes:

[0015] Inputting the virtual pathological training image and the real pathological training image into the discriminator to obtain a first discrimination result, and calculating a first loss value of the discriminator according to the first discrimination result; wherein, the first loss value is used to indicate the discrimination accuracy of the discriminator;

[0016] Adjusting the parameters of the discriminator according to the first loss value, and then return to execute the step of inputting the virtual pathological training image and the real pathological training image into the discriminator to obtain a first discrimination result, and calculating a first loss value of the discriminator according to the first discrimination result;

[0017] After the parameters of the discriminator are adjusted for the first preset number of times, input the virtual pathology training image and the real pathology training image into the discriminator to obtain a first discrimination result, and calculate a second loss value of the generator according to the first discrimination result; wherein, the second loss value is used to indicate the generation fidelity of the generator.

[0018] Adjust the parameters of the generator according to the second loss value, and return to execute the step of inputting the virtual pathology training image and the real pathology training image into the discriminator to obtain a first discrimination result, and calculating the second loss value of the generator according to the first discrimination result.

[0019] After the parameters of the generator are adjusted for the second preset number of times, return to execute the step of inputting the virtual pathology training image and the real pathology training image into the discriminator to obtain a first discrimination result, and calculating the first loss value of the discriminator according to the first discrimination result.

[0020] In one specific embodiment, the method further includes:

[0021] When the sum of the first loss value and the second loss value is less than a preset loss value, it is determined that the generative adversarial network converges.

[0022] In one specific embodiment, the fine-tuning of the converged generative adversarial network according to the second real imaging image and the real pathology image includes:

[0023] Obtain multiple groups of fine-tuning image sets of the target object; wherein, a group of fine-tuning image sets consists of a real imaging sub-image and a real pathology sub-image, the real imaging sub-image is a sub-image within the second real imaging image, the real pathology sub-image is a sub-image within the real pathology image, and the real imaging sub-image and the real pathology sub-image in the same group of fine-tuning image sets correspond to the same position within the lesion tissue sample of the target object.

[0024] Input a real imaging sub-image into the generator to obtain a virtual pathology sub-image of the generated lesion tissue sample.

[0025] Input the virtual pathological sub-image and the real pathological sub-image within the same set of fine-tuned image sets into the discriminator to obtain a second discrimination result. Iteratively adjust the parameters of the generator and the discriminator according to the second discrimination result to increase the probability of the discriminator making a correct discrimination and increase the probability of the discriminator making an incorrect discrimination on the image generated by the generator. Return to execute the step of inputting a real medical imaging sub-image into the generator to obtain the virtual pathological sub-image of the generated lesion tissue sample until all fine-tuned image sets are fine-tuned; wherein, the second discrimination result is used to indicate whether the virtual pathological sub-image and the real pathological sub-image are generated images.

[0026] In one specific embodiment, the method further includes:

[0027] Obtain multi-modal first real medical imaging images; wherein, the first real medical imaging images of different modalities are obtained by different medical imaging devices;

[0028] Generate the first real medical imaging images of each modality into virtual pathological images corresponding to the modality through the fine-tuned generative adversarial network.

[0029] In one specific embodiment, the method further includes:

[0030] Obtain the medical imaging three-dimensional stereoscopic structure data of the lesion tissue of the target object; wherein, the medical imaging three-dimensional stereoscopic structure data shows the lesion tissue from a medical imaging three-dimensional perspective, and the medical imaging three-dimensional stereoscopic structure data is composed of a plurality of first real medical imaging images of different two-dimensional planes;

[0031] Generate the plurality of first real medical imaging images of different two-dimensional planes into a plurality of virtual pathological images of different two-dimensional planes through the fine-tuned generative adversarial network, and reconstruct the plurality of virtual pathological images of different two-dimensional planes to obtain pathological three-dimensional stereoscopic structure data; wherein, the pathological three-dimensional stereoscopic structure data shows the lesion tissue from a pathological three-dimensional perspective.

[0032] An association device for medical imaging images and pathological images, the device includes:

[0033] A network fine-tuning module is configured to obtain a first true imaging image of a diseased tissue of a target object, a second true imaging image of a diseased tissue sample, and a true pathological image of the diseased tissue sample; wherein, the diseased tissue sample is obtained by performing core needle biopsy on the diseased tissue; and fine-tune a converged generative adversarial network according to the second true imaging image and the true pathological image; wherein, the fine-tuned generative adversarial network generates a virtual pathological image based on the true imaging image of the target object, and the virtual pathological image is similar to the true pathological image of the target object;

[0034] An association module is configured to generate, by using the fine-tuned generative adversarial network, the first true imaging image into a virtual pathological image of the diseased tissue.

[0035] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor is caused to execute the steps of the above-described method for associating an imaging image with a pathological image.

[0036] An apparatus for associating an imaging image with a pathological image includes a memory and a processor. The memory stores a computer program, and when the computer program is executed by the processor, the processor is caused to execute the steps of the above-described method for associating an imaging image with a pathological image.

[0037] The present invention provides a method, apparatus, device, and medium for associating an imaging image with a pathological image. First, a first true imaging image of a diseased tissue, a second true imaging image of a diseased tissue sample, and a true pathological image of the diseased tissue sample are obtained. Then, a converged generative adversarial network is fine-tuned according to the second true imaging image and the true pathological image until the generative adversarial network generates a virtual pathological image based on the true imaging image of the target object, and the virtual pathological image is similar to the true pathological image of the target object. That is, the generative adversarial network is enabled to have the ability to convert a true imaging image into a realistic virtual pathological image. Finally, the first true imaging image is generated into a virtual pathological image of the diseased tissue by using the fine-tuned generative adversarial network, so that a tight association between the imaging image and the pathological image of the diseased tissue can be achieved at the image level, promoting the associated research between the macroscopic and the microscopic. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present invention, and for those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0039] Wherein:

[0040] Figure 1 is a schematic flowchart of a training method for a generative adversarial network in an embodiment;

[0041] Figure 2 is a schematic flowchart of obtaining imaging images and pathological images after core needle biopsy in an embodiment;

[0042] Figure 3 is a schematic diagram of training a generative adversarial network in an embodiment;

[0043] Figure 4 is a schematic flowchart of iteratively tuning the parameters of the generator and discriminator according to the first discrimination result in an embodiment;

[0044] Figure 5 is a schematic flowchart of a correlation method between imaging images and pathological images in an embodiment;

[0045] Figure 6 is a schematic diagram of obtaining multiple sets of fine-tuning image sets of a target object in an embodiment;

[0046] Figure 7 is a schematic diagram of fine-tuning a generative adversarial network in an embodiment;

[0047] Figure 8 is a schematic diagram of a virtual pathological image of a diseased tissue in an embodiment;

[0048] Figure 9 is a schematic diagram of generating pathological three-dimensional structural data based on imaging three-dimensional structural data in an embodiment;

[0049] Figure 10 is a schematic structural diagram of a correlation device between imaging images and pathological images in an embodiment;

[0050] Figure 11 is a structural block diagram of a correlation device between imaging images and pathological images in an embodiment. Detailed implementation manners

[0051] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0052] To establish a close association at the image level between pathological images and imaging images, the present invention relies on a converged generative adversarial network, which includes a generator and a discriminator. The generator can generate virtual pathological images based on the real imaging images of different objects; and the virtual pathological images are very similar to the real pathological images, and can reflect the fine images at the microscopic cell level and tissue level at the same position compared with the real imaging images.

[0053] As Figure 1 shown, Figure 1 FIG. is a schematic flow chart of a training method for a generative adversarial network in an embodiment, and the generative adversarial network is made to converge based on this training method. The steps provided in this embodiment include:

[0054] Step 102, obtaining multiple sets of training image sets based on multiple training objects.

[0055] Specifically, randomly select multiple training objects, and the specific number is not limited. It can be understood that the more the number, the better the training effect. For one of the training objects, first, medical imaging devices such as Computed Tomography (CT) and Magnetic Resonance Imaging (MRI) can be used to take pictures outside the training object. Since the density, water content, and other parameters of the diseased tissue are usually different from those of the normal tissue, the diseased tissue usually has abnormal gray levels in the captured images. Based on this, the diseased tissue is located. And a core needle biopsy is performed at a certain position within the diseased tissue to obtain a strip-shaped tubular diseased tissue sample, and then through imaging devices such as CT / MRI, a real imaging training image is obtained. Then, the core needle biopsy diseased tissue sample is subjected to conventional chemical staining or immunostaining, and under a microscope or a digital pathology scanner, a real pathological training image is obtained. In this way, the obtained real imaging training image and the real pathological training image correspond to the same position within the diseased tissue sample of the same training object, and these two images are used as a set of training image sets. The overall process is as Figure 2 shown, where the lower image corresponds to the real imaging training image, and the upper right image corresponds to the real pathological training image. Similarly, core needle biopsies can be performed at other positions within the diseased tissue of this training object, and the same operations can be performed on all other training objects, so as to obtain multiple sets of training image sets.

[0056] Step 104, input a real imaging training image into the generator to obtain the generated virtual pathological training image.

[0057] In a specific embodiment, the generator adopts a UNet network structure, which is a fully convolutional network containing 4 layers of downsampling, skip connection structure, and 4 layers of upsampling. Its feature is that the convolutional layers are completely symmetric in the downsampling and upsampling parts, and the feature maps at the downsampling end can skip deep sampling and be spliced to the corresponding upsampling end. As Figure 3 shown, after inputting the real imaging training image into the generator, it is first the encoding process, that is, Encoder. The size of the real imaging training image is reduced through convolution and downsampling to extract some shallow features. Then, through the connection method of concat, the feature image obtained in the encoding stage is combined with the feature image obtained in the decoding stage to combine deep and shallow features and refine the image. Finally, it is the decoding process, that is, Decoder. Some deep features are obtained through convolution and upsampling. The convolution in the whole process adopts the valid padding method to ensure that the convolution results do not lack context features. In this embodiment, the image output after the last upsampling is the virtual pathology training image.

[0058] Step 106: Input the virtual pathology training image and the real pathology training image in the same set of training image sets into the discriminator to obtain a first discrimination result. According to the first discrimination result, iteratively adjust the parameters of the generator and the discriminator to increase the probability of the discriminator making a correct discrimination and increase the probability of the image generated by the generator making the discriminator make a wrong discrimination. Return to execute step 104 until the generative adversarial network converges.

[0059] Among them, the first discrimination result is used to indicate whether the virtual pathology training image and the real pathology training image are generated images, that is, to indicate whether the input image is a real image obtained by shooting or a virtual generated fake image.

[0060] As Figure 3 shown, input the virtual pathology training image and the real pathology training image in the same set of training image sets into the discriminator. In a specific embodiment, the discriminator mainly adopts a Resnet50 network structure. The ResNet50 network architecture includes the initial convolutional pooling layer, the final pooling fully connected layer, and 16 residual modules. Each residual module contains two convolutional layers, a normalization layer, a ReLU activation layer, and a residual link. If the Resnet50 network has reached the optimum and the network is further deepened, the residual mapping output by the convolutional layer will be pushed to 0, and only the identity mapping output by the residual link remains. In this way, theoretically, the network is always in the optimum state, and the performance of the network will not decrease as the depth increases. The final pooling fully connected layer of Resnet50, and then determine the first discrimination result by adding a softmax layer that generates a one-dimensional output.

[0061] Specifically, as Figure 4 shown, the process of iteratively adjusting the parameters of the generator and discriminator according to the first discrimination result includes:

[0062] Step 406a: Input the virtual pathology training images and real pathology training images into the discriminator to obtain the first discrimination result, and calculate the first loss value of the discriminator according to the first discrimination result.

[0063] Among them, the first loss value is used to indicate the discrimination accuracy of the discriminator. The larger the first loss value, the less accurate the discrimination of the discriminator; conversely, the more accurate the discrimination of the discriminator. The calculation formula for calculating this first loss value is:

[0064] Loss D = -[ylog y^+(1 - y)log(1 - y^)]

[0065] where y is the true label of the input image, which is 0 if it is a virtual pathology sub-image and 1 if it is a real pathology sub-image, and y^ represents the probability vector output by the discriminator.

[0066] Step 406b: Adjust the parameters of the discriminator according to the first loss value, and return to execute Step 406a.

[0067] That is, first fix the parameters of the generator, adjust the parameters of the discriminator based on the calculated first loss value, then input another real pathology training image and the corresponding virtual pathology training image into the discriminator, and adjust the parameters of the discriminator again based on the calculated first loss value, and so on in a loop.

[0068] Specifically, the adaptive moment estimation algorithm (adam) can be used to adjust the parameters in the discriminator to reduce the first loss value. For example, the number of iterations of the adam algorithm is set to m0 times in total, the initial learning rate is set to 0.001, the weight decay is set to 0.0005, and every m1 (m1 < m0) iterations, the learning rate decays to 1 / 10 of the original.

[0069] Step 406c: After the parameters of the discriminator are adjusted for the first preset number of times, input the virtual pathology training images and real pathology training images into the discriminator to obtain the first discrimination result, and calculate the second loss value of the generator according to the first discrimination result.

[0070] Among them, the second loss value is used to indicate the generation fidelity of the generator. The larger the second loss value, the less realistic the image generated by the generator; conversely, the more realistic the image generated by the generator. The calculation formula for calculating the second loss value is:

[0071] Loss G = ||I r - If || 1

[0072] Among them, I r indicates the real pathological sub-image, and I f indicates the virtual pathological sub-image.

[0073] Step 406d, adjust the parameters of the generator according to the second loss value, and return to execute step 406c.

[0074] That is, after the parameters of the discriminator are adjusted for the first preset number of times, the parameters of the discriminator are fixed, and the parameters of the generator are adjusted based on the calculated second loss value. Then, another real pathological training image and the corresponding virtual pathological training image are input into the discriminator, and the parameters of the generator are adjusted based on the calculated second loss value, and this process is repeated continuously.

[0075] Specifically, the adam algorithm can be used to adjust the parameters in the generator to reduce the first loss value. For example, the number of iterations of the adam algorithm is set to n0 times in total, the initial learning rate is set to 0.001, the weight decay is set to 0.0005, and every n1 (n1 < n0) iterations, the learning rate decays to 1 / 10 of the original.

[0076] Step 406e, after the parameters of the generator are adjusted for the second preset number of times, return to execute step 406a. When the sum of the first loss value and the second loss value is less than the preset loss value, it is determined that the generative adversarial network converges.

[0077] That is, after the parameters of the generator are adjusted for the second preset number of times, the parameters of the generator are fixed again, and the parameters of the discriminator are adjusted based on the calculated first loss value. After the first preset number of adjustments, the parameters of the discriminator are fixed, and the parameters of the generator are adjusted based on the calculated second loss value, and this process is repeated continuously. Until Loss D +Loss G < L (preset loss value), it is determined that the generative adversarial network converges.

[0078] Through the above training method of the generative adversarial network, the generative adversarial network can be made to converge, so that the conversion from imaging images to pathological images can be realized.

[0079] Such as Figure 5 shown, Figure 5 is a schematic flowchart of the method for associating imaging images and pathological images in an embodiment, which is applied to the above-converged generative adversarial network. The steps provided by the method for associating imaging images and pathological images in this embodiment include:

[0080] Step 502: Obtain the first true imaging image of the diseased tissue of the target object, the second true imaging image of the diseased tissue sample, and the true pathological image of the diseased tissue sample.

[0081] For the target object, use a certain imaging device, such as a CT device, to take pictures outside the target object, locate the diseased tissue of the target object, and obtain the first true imaging image of the target object at the diseased tissue; as Figure 2 shown, then perform a core needle biopsy at a certain position within the diseased tissue to obtain a strip-shaped tubular diseased tissue sample, and then use the same CT device to obtain a second true imaging image. Then, after the core needle biopsy diseased tissue sample undergoes conventional chemical staining or immunostaining, a true pathological image is obtained under a microscope or a digital pathology scanner.

[0082] It can be understood that the first true imaging image is a global image of the diseased tissue, while the second true imaging image is a local image of the diseased tissue sample.

[0083] Step 504: Fine-tune the converged generative adversarial network according to the second true imaging image and the true pathological image.

[0084] It can be understood that the second true imaging image and the true pathological image correspond to the same position within the diseased tissue sample of the target object. Based on these two images, fine-tune the converged generative adversarial network so that the generative adversarial network can be adapted to the target object. The fine-tuned generative adversarial network can generate a virtual pathological image based on the true imaging image of the target object, and the virtual pathological image is similar to the true pathological image of the target object.

[0085] In one specific embodiment, the process of fine-tuning is as follows:

[0086] First, obtain multiple sets of fine-tuning image sets of the target object.

[0087] As Figure 6 shown, the upper one is the second true imaging image, and the lower one is the true pathological image. Select a sub-image as the true imaging sub-image in the square position of the second true imaging image, and at the same time select a sub-image as the true pathological sub-image in the square position at the same position within the true pathological image, so as to obtain a set of fine-tuning image sets. Similarly, another set of fine-tuning image sets can be obtained at the triangular position corresponding to the same position, and repeat such operations until the required multiple sets of fine-tuning image sets are obtained.

[0088] Second, input a true imaging sub-image into the generator to obtain the generated virtual pathological sub-image of the diseased tissue sample.

[0089] Thirdly, input the virtual pathological sub-images and the real pathological sub-images within the same set of fine-tuned image sets into the discriminator to obtain a second discrimination result. According to the second discrimination result, iteratively adjust the parameters of the generator and the discriminator to increase the probability of the discriminator making a correct discrimination and increase the probability that the images generated by the generator cause the discriminator to make an incorrect discrimination. Return to execute the step of inputting a real radiological sub-image into the generator to obtain the virtual pathological sub-image of the generated lesion tissue sample until all the fine-tuned image sets are fine-tuned.

[0090] Among them, the second discrimination result is used to indicate whether the virtual pathological sub-images and the real pathological sub-images are generated images. The process of this fine-tuning is as Figure 7 shown and is similar to step 106, except that the fine-tuning stops after all the fine-tuned image sets are input into the generative adversarial network, and the specific process will not be elaborated here.

[0091] Step 506, generate the virtual pathological image of the lesion tissue from the first real radiological image through the fine-tuned generative adversarial network.

[0092] Specifically, after inputting the first real radiological image into the generator, the virtual pathological image of the lesion tissue as shown in Figure 8 can be obtained.

[0093] For the above-mentioned method for associating radiological images and pathological images, first obtain the first real radiological image of the lesion tissue, the second real radiological image of the lesion tissue sample, and the real pathological image of the lesion tissue sample. Then, fine-tune the converged generative adversarial network according to the second real radiological image and the real pathological image until the generative adversarial network generates a virtual pathological image based on the real radiological image of the target object, and this virtual pathological image is similar to the real pathological image of the target object. That is, enable the generative adversarial network to have the ability to convert real radiological images into realistic virtual pathological images. Finally, generate the virtual pathological image of the lesion tissue from the first real radiological image through the fine-tuned generative adversarial network, so that for the lesion tissue, the radiological images and pathological images can be closely associated at the image level, promoting the correlative research between the macroscopic and microscopic aspects.

[0094] In the previous embodiment, only the first real radiological image in one modality (based on CT equipment) was obtained. Of course, it is also possible to form multi-modal first real radiological images with different parameters based on other radiological equipment such as MRI, electrocardiogram instruments, electroencephalogram instruments, etc.; and then based on the process of steps 502 - 506, realize generating the virtual pathological images corresponding to each modality from the first real radiological images of each modality through the fine-tuned generative adversarial network. In this way, the specific application scenarios of the present invention can be fully expanded and can be applied to various radiological equipment.

[0095] Since the association between the above images is limited to a two-dimensional surface, and the image information of a two-dimensional surface is limited, in order to further strengthen the association between the pathological image and the imaging image, in a specific embodiment, the following steps are further performed:

[0096] First, the imaging three-dimensional structural data of the target object's diseased tissue is obtained.

[0097] The imaging three-dimensional structure data is to display the diseased tissue from the three-dimensional perspective of imaging. Specifically, the imaging three-dimensional structure data can be constructed by scanning two-dimensional surfaces with multiple layers. For example, Figure 9 As shown, the imaging three-dimensional structure data is the first real imaging image of the two-dimensional surface formed by the X-axis and the Y-axis, which is continuously extended along the Z-axis. Therefore, the Z-axis can be divided into m layers of the first real imaging image of the two-dimensional surface formed by the X-axis and the Y-axis.

[0098] Second, a plurality of first real imaging images of different two-dimensional surfaces are generated into a plurality of virtual pathological images of different two-dimensional surfaces through a fine-tuned generative adversarial network, and the plurality of virtual pathological images of different two-dimensional surfaces are reconstructed to obtain pathological three-dimensional structural data.

[0099] That is, the first real imaging image of each two-dimensional surface is input into the generative adversarial network in the previous embodiment according to the order between layers, so that multiple virtual pathological images of different two-dimensional surfaces can be generated and reconstructed to obtain the following: Figure 9 The pathological three-dimensional structural data shown displays the diseased tissue from a pathological three-dimensional perspective.

[0100] It is worth emphasizing that we should start from the practical value. Many advanced diseased tissues cannot be surgically removed, and pathological samples can only be obtained through coarse needle puncture biopsy, small samples can be collected in vivo, and pathological and imaging images of small samples can be collected in vitro. Large samples cannot be surgically removed, and more pathological images cannot be obtained. Therefore, based on the above method, it is possible to achieve the derivation and transformation from lines to surfaces and from surfaces to three-dimensional images of living bodies. For cases that cannot be surgically removed, three-dimensional pathological images are constructed to better guide the improvement of the condition of advanced diseased tissue objects.

[0101] For early-stage patients, the diseased tissue has been completely removed, and the prognosis is often good, and no further intervention is needed. Therefore, under the condition of taking a small amount of biological samples from living bodies, relying on the correspondence between the pathology and imaging of a small amount of tissue, the three-dimensional pathological images of the living diseased tissue can be derived, which has greater clinical value.

[0102] In one embodiment, as Figure 10 shown, an apparatus for associating imaging images with pathological images is proposed. The apparatus includes:

[0103] A network fine-tuning module 1002, configured to obtain a first real imaging image of a diseased tissue of a target object, a second real imaging image of a diseased tissue sample, and a real pathological image of the diseased tissue sample; wherein, the diseased tissue sample is obtained by performing core needle biopsy on the diseased tissue; and fine-tuning a converged generative adversarial network according to the second real imaging image and the real pathological image; wherein, the fine-tuned generative adversarial network generates a virtual pathological image based on the real imaging image of the target object, and the virtual pathological image is similar to the real pathological image of the target object;

[0104] An association module 1004, configured to generate a virtual pathological image of the diseased tissue from the first real imaging image through the fine-tuned generative adversarial network.

[0105] Figure 11 shows an internal structural diagram of an apparatus for associating imaging images with pathological images in one embodiment. As Figure 11 shown, the apparatus for associating imaging images with pathological images includes a processor, a memory, and a network interface connected through a system bus. Among them, the memory includes a non-volatile storage medium and an internal memory. The non-volatile storage medium of the apparatus for associating imaging images with pathological images stores an operating system, and may also store a computer program, which when executed by the processor, can enable the processor to implement the method for associating imaging images with pathological images. The internal memory may also store a computer program, which when executed by the processor, can enable the processor to execute the method for associating imaging images with pathological images. Those skilled in the art can understand that Figure 11 the structure shown in

[0106] An imaging image and pathology image association device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the following steps are implemented: obtaining a first true imaging image of a diseased tissue of a target object, a second true imaging image of a diseased tissue sample, and a true pathology image of the diseased tissue sample; wherein, the diseased tissue sample is obtained by performing core needle biopsy on the diseased tissue; fine-tuning a converged generative adversarial network according to the second true imaging image and the true pathology image; wherein, the fine-tuned generative adversarial network generates a virtual pathology image based on the true imaging image of the target object, and the virtual pathology image is similar to the true pathology image of the target object; generating a virtual pathology image of the diseased tissue from the first true imaging image through the fine-tuned generative adversarial network.

[0107] A computer-readable storage medium storing a computer program, which when executed by a processor, implements the following steps: obtaining a first true imaging image of a diseased tissue of a target object, a second true imaging image of a diseased tissue sample, and a true pathology image of the diseased tissue sample; wherein, the diseased tissue sample is obtained by performing core needle biopsy on the diseased tissue; fine-tuning a converged generative adversarial network according to the second true imaging image and the true pathology image; wherein, the fine-tuned generative adversarial network generates a virtual pathology image based on the true imaging image of the target object, and the virtual pathology image is similar to the true pathology image of the target object; generating a virtual pathology image of the diseased tissue from the first true imaging image through the fine-tuned generative adversarial network.

[0108] It should be noted that the above imaging image and pathology image association method, device, equipment, and computer-readable storage medium belong to a general inventive concept, and the content in the embodiments of the imaging image and pathology image association method, device, equipment, and computer-readable storage medium can be mutually applicable.

[0109] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. This program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, storage, database, or other medium used in the embodiments provided in the present application can include non-volatile and / or volatile memories. Non-volatile memories can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memories can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and Rambus dynamic RAM (RDRAM), etc.

[0110] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope described in this specification.

[0111] The above embodiments only represent several implementation manners of the present application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the patent scope of the present application. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can still be made, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application should be subject to the appended claims.

Claims

1. A method for associating imaging images with pathological images, characterized in that, the method includes: Obtaining a first real imaging image of a diseased tissue of a target object, a second real imaging image of a diseased tissue sample, and a real pathological image of the diseased tissue sample; wherein, the diseased tissue sample is obtained by performing core needle biopsy on the diseased tissue; Fine-tuning a converged generative adversarial network according to the second real imaging image and the real pathological image; wherein, the fine-tuned generative adversarial network generates a virtual pathological image based on the real imaging image of the target object, and the virtual pathological image is similar to the real pathological image of the target object; Generating the first real imaging image into a virtual pathological image of the diseased tissue through the fine-tuned generative adversarial network; Wherein, the generative adversarial network includes a generator and a discriminator, and the method further includes: Obtaining multiple sets of training image sets based on multiple training objects; wherein, a set of training image sets consists of a real imaging training image and a real pathological training image, and the real imaging training image and the real pathological training image in the same set of training image sets correspond to the same position in the diseased tissue sample of the same training object; Inputting a real imaging training image into the generator to obtain a generated virtual pathological training image; Inputting the virtual pathological training image and the real pathological training image in the same set of training image sets into the discriminator to obtain a first discrimination result, and iteratively adjusting the parameters of the generator and the discriminator according to the first discrimination result to increase the probability that the discriminator discriminates correctly, and increasing the probability that the image generated by the generator causes the discriminator to make a wrong discrimination, and returning to execute the step of inputting a real imaging training image into the generator to obtain a generated virtual pathological training image until the generative adversarial network converges; wherein, the first discrimination result is used to indicate whether the virtual pathological training image and the real pathological training image are generated images.

2. The method according to claim 1, characterized in that, the step of inputting the virtual pathological training image and the real pathological training image in the same set of training image sets into the discriminator to obtain a first discrimination result, and iteratively adjusting the parameters of the generator and the discriminator according to the first discrimination result includes: Inputting the virtual pathological training image and the real pathological training image into the discriminator to obtain a first discrimination result, and calculating a first loss value of the discriminator according to the first discrimination result; wherein, the first loss value is used to indicate the discrimination accuracy of the discriminator; Adjusting the parameters of the discriminator according to the first loss value, and returning to execute the step of inputting the virtual pathological training image and the real pathological training image into the discriminator to obtain a first discrimination result, and calculating the first loss value of the discriminator according to the first discrimination result; After the parameters of the discriminator are adjusted for the first preset number of times, input the virtual pathology training image and the real pathology training image into the discriminator to obtain a first discrimination result, and calculate a second loss value of the generator according to the first discrimination result; wherein, the second loss value is used to indicate the generation fidelity of the generator. Adjust the parameters of the generator according to the second loss value, and return to execute the step of inputting the virtual pathology training image and the real pathology training image into the discriminator to obtain a first discrimination result, and calculating the second loss value of the generator according to the first discrimination result. After the parameters of the generator are adjusted for the second preset number of times, return to execute the step of inputting the virtual pathology training image and the real pathology training image into the discriminator to obtain a first discrimination result, and calculating the first loss value of the discriminator according to the first discrimination result.

3. The method according to claim 2, wherein, the method further includes: When the sum of the first loss value and the second loss value is less than a preset loss value, it is determined that the generative adversarial network converges.

4. The method according to claim 1, wherein, the fine-tuning of the converged generative adversarial network according to the second real imaging image and the real pathology image includes: Obtain multiple groups of fine-tuning image sets of the target object; wherein, a group of fine-tuning image sets consists of a real sub-imaging image and a real sub-pathology image, the real sub-imaging image is a sub-image within the second real imaging image, the real sub-pathology image is a sub-image within the real pathology image, and the real sub-imaging image and the real sub-pathology image in the same group of fine-tuning image sets correspond to the same position within the lesion tissue sample of the target object. Input a real sub-imaging image into the generator to obtain a virtual sub-pathology image of the generated lesion tissue sample. Input the virtual sub-pathology image and the real sub-pathology image in the same group of fine-tuning image sets into the discriminator to obtain a second discrimination result, and perform iterative parameter adjustment on the generator and the discriminator according to the second discrimination result to increase the probability of the discriminator making a correct discrimination, and increase the probability that the image generated by the generator causes the discriminator to make a wrong discrimination, and return to execute the step of inputting a real sub-imaging image into the generator to obtain a virtual sub-pathology image of the generated lesion tissue sample until all fine-tuning image sets are fine-tuned; wherein, the second discrimination result is used to indicate whether the virtual sub-pathology image and the real sub-pathology image are generated images.

5. The method according to claim 1, wherein, the method further includes: Obtain a multi-modal first real imaging image; wherein, the first real imaging images of different modalities are obtained by different imaging devices. Generate virtual pathology images corresponding to each modality from each modality of the first real imaging image through the fine-tuned generative adversarial network.

6. The method according to claim 1, wherein, The method further includes: Obtaining three-dimensional stereoscopic structure data of the diseased tissue of the target object; wherein, the three-dimensional stereoscopic structure data of the imaging shows the diseased tissue from a three-dimensional perspective of imaging, and the three-dimensional stereoscopic structure data of the imaging is composed of a plurality of first real imaging images of different two-dimensional planes; Generating, by the fine-tuned generative adversarial network, the first real imaging images of the plurality of different two-dimensional planes into virtual pathological images of the plurality of different two-dimensional planes, and reconstructing the virtual pathological images of the plurality of different two-dimensional planes to obtain three-dimensional stereoscopic structure data of pathology; wherein, the three-dimensional stereoscopic structure data of pathology shows the diseased tissue from a three-dimensional perspective of pathology.

7. An apparatus for associating imaging images and pathological images, Characterized in that The apparatus includes: A network fine-tuning module, configured to obtain a first real imaging image of the diseased tissue of the target object, a second real imaging image of the diseased tissue sample, and a real pathological image of the diseased tissue sample; wherein, the diseased tissue sample is obtained by performing core needle biopsy on the diseased tissue; and fine-tuning the converged generative adversarial network according to the second real imaging image and the real pathological image; wherein, the fine-tuned generative adversarial network generates a virtual pathological image based on the real imaging image of the target object, and the virtual pathological image is similar to the real pathological image of the target object; An association module, configured to generate, by the fine-tuned generative adversarial network, the first real imaging image into a virtual pathological image of the diseased tissue; Wherein, the generative adversarial network includes a generator and a discriminator, and the apparatus is further configured to: obtain multiple sets of training image sets based on multiple training objects; wherein, a set of training image sets consists of a real imaging training image and a real pathological training image, and the real imaging training image and the real pathological training image in the same set of training image sets correspond to the same position in the diseased tissue sample of the same training object; input a real imaging training image into the generator to obtain a generated virtual pathological training image; input the virtual pathological training image and the real pathological training image in the same set of training image sets into the discriminator to obtain a first discrimination result, and perform iterative parameter tuning on the generator and the discriminator according to the first discrimination result to increase the probability of the discriminator making a correct discrimination, and increase the probability of the image generated by the generator making the discriminator make a wrong discrimination, and return to execute the step of inputting a real imaging training image into the generator to obtain a generated virtual pathological training image until the generative adversarial network converges; wherein, the first discrimination result is used to indicate whether the virtual pathological training image and the real pathological training image are generated images.

8. A computer-readable storage medium storing a computer program, Characterized in that When the computer program is executed by a processor, the processor is caused to execute the steps of the method according to any one of claims 1 to 6.

9. An association device for imaging images and pathological images, comprising a memory and a processor, characterized in that, the memory stores a computer program, and when the computer program is executed by the processor, the processor is caused to execute the steps of the method according to any one of claims 1 to 6.

Images