A processing method and device for three-dimensional lesion localization based on multimodal imaging

Abstract

The embodiment of the present invention relates to a processing method and device for three-dimensional lesion positioning based on multimodal images, the method comprising: receiving multimodal images; performing image noise reduction and artifact removal on each modal image; performing spatial registration on each modal preprocessing image based on a reference modal image; performing feature extraction of the corresponding modality on each modal preprocessing image; performing feature fusion on the feature map and registration map corresponding to each modal preprocessing image; performing full-modal feature fusion on all obtained modal fusion feature maps; performing voxel-level organ tissue and lesion semantic segmentation on the full-modal feature map; performing three-dimensional organ tissue structure modeling according to the full-modal feature map, the organ tissue semantic map and the lesion semantic map, and performing lesion area positioning marking on the modeling structure. The present invention can improve the recognition accuracy and precision of the lesion area.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a processing method and device for three-dimensional lesion positioning based on multimodal images. Background Art

[0002] Before performing surgery on a patient (or during the operation), the doctor will locate and identify the patient's lesion area based on a series of medical images and set or modify the surgical plan based on the identification results. These medical images commonly include three-dimensional magnetic resonance imaging (MRI), computed tomography (CT), positron emission tomography (PET), two-dimensional X-ray (Xray), ultrasound (US), etc. At present, when doctors locate and identify the lesion area based on part or all of the above medical images, most of them use manual methods to identify the lesion area, that is, based on the personal or team medical imaging experience of the doctor or doctor team. Due to the limitation of human factors, this manual identification method generally has problems such as low recognition accuracy and unstable recognition accuracy. Summary of the invention

[0003] The purpose of the present invention is to provide a processing method, device, electronic device and computer-readable storage medium for three-dimensional lesion positioning based on multimodal images in view of the defects of the prior art. After receiving the multimodal images, the present invention first performs image denoising and artifact removal processing on each modality image respectively, then performs spatial registration and self-modal feature extraction on each modality preprocessing map, and fuses the spatial registration feature and the self-modal feature to obtain the corresponding single-modal fusion feature map, then performs full-modal feature fusion on all single-modal fusion feature maps to obtain the corresponding full-modal feature map, then performs voxel-level organ tissue and lesion semantic segmentation on the full-modal feature map to obtain the corresponding organ tissue semantic map and lesion semantic map, and then performs three-dimensional organ tissue structure modeling according to the full-modal feature map, the organ tissue semantic map and the lesion semantic map, and performs lesion area positioning marking on the modeling structure. Through the present invention, on the one hand, the purpose of improving the accuracy of lesion area recognition can be achieved through multimodal feature fusion, and on the other hand, the purpose of improving the accuracy of lesion area recognition can be achieved through voxel-level semantic segmentation.

[0004] To achieve the above-mentioned purpose, a first aspect of an embodiment of the present invention provides a processing method for three-dimensional lesion localization based on multimodal images, the method comprising:

[0005] Receiving a plurality of first modality images; each of the first modality images corresponds to a modality type; the modality types include three-dimensional MRI images, three-dimensional CT images, three-dimensional PET images, two-dimensional X-ray images, and two-dimensional US images;

[0006] Performing image denoising and artifact removal processing on each of the first modality images to obtain a corresponding first modality preprocessing image; the three-dimensional or two-dimensional size of each of the first modality preprocessing images is consistent with the three-dimensional or two-dimensional size of the corresponding first modality image;

[0007] The first modality preprocessing image whose modality type matches the preset reference image type is recorded as a corresponding reference modality image; and each of the first modality preprocessing images is spatially registered based on the reference modality image to obtain a corresponding first modality registration image; the reference image type includes a three-dimensional MRI image and a three-dimensional CT image; the three-dimensional or two-dimensional size of each of the first modality registration images is consistent with the three-dimensional or two-dimensional size of the corresponding first modality image;

[0008] Performing feature extraction processing of the corresponding modality on each of the first modality preprocessing images to obtain a corresponding first modality feature image; the three-dimensional or two-dimensional size of each of the first modality feature images is consistent with the three-dimensional or two-dimensional size of the corresponding first modality image;

[0009] Performing feature fusion on the first modality feature map and the first modality registration map corresponding to each of the first modality preprocessing maps in a feature channel splicing manner to obtain a corresponding first modality fusion feature map;

[0010] Performing full-modal feature fusion on all the first modal fusion feature maps obtained to obtain corresponding full-modal feature maps;

[0011] Performing voxel-level organ tissue and lesion semantic segmentation on the full-modality feature map to obtain corresponding organ tissue semantic map and lesion semantic map;

[0012] A three-dimensional organ tissue structure model is performed based on the full modality feature map, the organ tissue semantic map and the lesion semantic map, and the lesion area positioning mark is performed on the modeled structure.

[0013] Preferably, the first modality image whose modality type is a three-dimensional MRI image includes a plurality of first voxel points; the first voxel points include first voxel coordinates and a first MRI feature vector; the first MRI feature vector is composed of one or more MRI image feature data;

[0014] The first modality image whose modality type is a three-dimensional CT image includes a plurality of second voxel points; the second voxel points include second voxel coordinates and a first CT feature vector; the first CT feature vector is composed of one or more CT image feature data;

[0015] The first modality image whose modality type is a three-dimensional PET image includes a plurality of third voxel points; the third voxel points include third voxel coordinates and a first PET feature vector; the first PET feature vector is composed of one or more PET image feature data;

[0016] The first modality image whose modality type is a two-dimensional Xray image includes a plurality of first pixel points; the first pixel points include first pixel coordinates and a first Xray feature vector; the first Xray feature vector is composed of one or more Xray image feature data;

[0017] The first modality image whose modality type is a two-dimensional US image includes a plurality of second pixel points; the second pixel points include second pixel coordinates and a first US feature vector; the first US feature vector is composed of one or more US image feature data;

[0018] The first modality registration map corresponding to the first modality image whose modality type is a three-dimensional MRI image, a three-dimensional CT image or a three-dimensional PET image includes a plurality of fourth voxel points; the fourth voxel points correspond one-to-one to the first, second or third voxel points of the corresponding first modality image; the fourth voxel points include fourth voxel coordinates and a first registration feature vector; the first registration feature vector is composed of a first rotation feature vector and a first displacement feature vector; the first rotation feature vector is composed of matrix parameters of a three-dimensional rotation matrix; the first displacement feature vector is composed of matrix parameters of a three-dimensional displacement matrix;

[0019] The first modality feature map corresponding to the first modality image whose modality type is a three-dimensional MRI image, a three-dimensional CT image, or a three-dimensional PET image includes a plurality of fifth voxel points; the fifth voxel points correspond one-to-one to the first, second, or third voxel points of the corresponding first modality image; the fifth voxel points include fifth voxel coordinates and a first image feature vector;

[0020] The first modality registration map corresponding to the first modality image whose modality type is a two-dimensional Xray image or a two-dimensional US image includes a plurality of third pixel points; the third pixel points correspond one-to-one to the first or second pixel points of the corresponding first modality image; the third pixel points include third pixel coordinates and a second registration feature vector; the second registration feature vector is composed of a second rotation feature vector and a second displacement feature vector; the second rotation feature vector is composed of matrix parameters of a two-dimensional rotation matrix; the second displacement feature vector is composed of matrix parameters of a two-dimensional displacement matrix;

[0021] The first modality feature map corresponding to the first modality image whose modality type is a two-dimensional Xray image or a two-dimensional US image includes a plurality of fourth pixel points; the fourth pixel points correspond one-to-one to the first or second pixel points of the corresponding first modality image; the fourth pixel points include fourth pixel coordinates and a second image feature vector;

[0022] The first modality fusion feature map corresponding to the first modality image whose modality type is a three-dimensional MRI image, a three-dimensional CT image, or a three-dimensional PET image includes a plurality of sixth voxel points; the sixth voxel points correspond one-to-one to the first, second, or third voxel points of the corresponding first modality image; the sixth voxel points include sixth voxel coordinates and a first fusion feature vector; the first fusion feature vector is composed of the corresponding first image feature vector and the first registration feature vector;

[0023] The first modality fusion feature map corresponding to the first modality image whose modality type is a two-dimensional Xray image or a two-dimensional US image includes multiple fifth pixel points; the fifth pixel points correspond one-to-one to the first or second pixel points of the corresponding first modality image; the fifth pixel points include fifth pixel coordinates and a second fusion feature vector; the second fusion feature vector is composed of the corresponding second image feature vector and the second registration feature vector.

[0024] Preferably, the spatially registering each of the first modal preprocessing images based on the reference modal image to obtain a corresponding first modal registration image specifically includes:

[0025] Using each of the first modal preprocessing graphs as the corresponding current modal preprocessing graph;

[0026] and taking the modality type corresponding to the current modality preprocessing graph as the corresponding current floating modality type;

[0027] and identifying the reference image type and the current floating modality type;

[0028] If the reference image type is the same as the current floating modality type, constructing a first modality registration image whose three-dimensional size is consistent with the three-dimensional size of the current modality preprocessing image; and uniformly setting all the first rotation feature vectors of the first modality registration image currently constructed based on a preset zero rotation feature vector; and uniformly setting all the first displacement feature vectors of the first modality registration image currently constructed based on a preset zero displacement feature vector;

[0029] If the reference image type and the current floating modality type are the corresponding three-dimensional MRI image and three-dimensional CT image, respectively, the MRI reference-CT floating registration model in the preset registration model library is used as the corresponding current registration model; and the reference modality image and the current modality preprocessing image are input into the current registration model as the corresponding MRI reference image and CT floating image for configuration processing to obtain the corresponding first modality registration image;

[0030] If the reference image type and the current floating modality type are the corresponding three-dimensional MRI image and three-dimensional PET image respectively, the MRI reference-PET floating registration model in the registration model library is used as the corresponding current registration model; and the reference modality image and the current modality pre-processing image are input into the current registration model as the corresponding MRI reference image and PET floating image for configuration processing to obtain the corresponding first modality registration image;

[0031] If the reference image type and the current floating modality type are corresponding three-dimensional MRI images and two-dimensional Xray images, respectively, the MRI reference-Xray floating registration model in the registration model library is used as the corresponding current registration model; and the reference modality image and the current modality preprocessing image are input into the current registration model as the corresponding MRI reference image and Xray floating image for configuration processing to obtain the corresponding first modality registration image;

[0032] If the reference image type and the current floating modality type are corresponding three-dimensional MRI images and two-dimensional US images, respectively, the MRI reference-US floating registration model in the registration model library is used as the corresponding current registration model; and the reference modality image and the current modality pre-processing image are input into the current registration model as the corresponding MRI reference image and US floating image for configuration processing to obtain the corresponding first modality registration image;

[0033] If the reference image type and the current floating modality type are corresponding three-dimensional CT images and three-dimensional MRI images, respectively, a CT reference-MRI floating registration model in a preset registration model library is used as the corresponding current registration model; and the reference modality image and the current modality preprocessing image are input into the current registration model as the corresponding CT reference image and MRI floating image for configuration processing to obtain the corresponding first modality registration image;

[0034] If the reference image type and the current floating modality type are corresponding three-dimensional CT images and three-dimensional PET images, respectively, a CT reference-PET floating registration model in a preset registration model library is used as the corresponding current registration model; and the reference modality image and the current modality preprocessing image are input into the current registration model as the corresponding CT reference image and PET floating image for configuration processing to obtain the corresponding first modality registration image;

[0035] If the reference image type and the current floating modality type are corresponding three-dimensional CT images and two-dimensional Xray images, respectively, the CT reference-Xray floating registration model in the preset registration model library is used as the corresponding current registration model; and the reference modality map and the current modality preprocessing map are input into the current registration model as the corresponding CT reference map and Xray floating map for configuration processing to obtain the corresponding first modality registration map;

[0036] If the reference image type and the current floating modality type are corresponding three-dimensional CT images and two-dimensional US images, respectively, a CT reference-US floating registration model in a preset registration model library is used as the corresponding current registration model; and the reference modality image and the current modality preprocessing image are input into the current registration model as the corresponding CT reference image and US floating image for configuration processing to obtain the corresponding first modality registration image;

[0037] And the first modality registration image obtained this time is output as the spatial registration result of the current modality preprocessing image.

[0038] Furthermore, the registration model library includes the MRI benchmark-CT floating registration model, the MRI benchmark-PET floating registration model, the MRI benchmark-Xray floating registration model, the MRI benchmark-US floating registration model, the CT benchmark-MRI floating registration model, the CT benchmark-PET floating registration model, the CT benchmark-Xray floating registration model and the CT benchmark-US floating registration model; each registration model in the registration model library is implemented based on a deep learning neural network model.

[0039] Preferably, performing feature extraction processing of the corresponding modality on each of the first modal preprocessing graphs to obtain the corresponding first modal feature graph specifically includes:

[0040] Using each of the first modal preprocessing graphs as the corresponding current modal preprocessing graph;

[0041] and taking the modality type corresponding to the current modality preprocessing graph as the corresponding current modality type;

[0042] And identify the current modality type; if the current modality type is a three-dimensional MRI image, the MRI image feature extraction model in the preset feature extraction model library is used as the corresponding current feature extraction model; if the current modality type is a three-dimensional CT image, the CT image feature extraction model in the feature extraction model library is used as the corresponding current feature extraction model; if the current modality type is a three-dimensional PET image, the PET image feature extraction model in the feature extraction model library is used as the corresponding current feature extraction model; if the current modality type is a two-dimensional Xray image, the Xray image feature extraction model in the feature extraction model library is used as the corresponding current feature extraction model; if the current modality type is a two-dimensional US image, the US image feature extraction model in the feature extraction model library is used as the corresponding current feature extraction model;

[0043] And based on the current feature extraction model, feature extraction processing is performed on the current modal preprocessing image to obtain the corresponding first modal feature image.

[0044] Furthermore, the feature extraction model library includes the MRI image feature extraction model, the CT image feature extraction model, the PET image feature extraction model, the Xray image feature extraction model and the US image feature extraction model; each feature extraction model of the feature extraction model library is implemented based on a class of convolutional neural network models, a class of residual network models or a class of densely connected convolutional network models.

[0045] Preferably, performing full-modal feature fusion on all the first modal fusion feature maps obtained to obtain corresponding full-modal feature maps specifically includes:

[0046] The first modal fusion feature map corresponding to the reference image type is used as the corresponding reference modal fusion feature map; and a three-dimensional space is planned as the corresponding fusion coding space based on the three-dimensional size of the reference modal fusion feature map; and a preset full modal coding model is used to perform full modal feature fusion coding processing in the fusion coding space according to all the first modal fusion feature maps to obtain the corresponding full modal feature map; the full modal coding model is implemented based on a type of multimodal autoencoder model or a type of multimodal Transformer model.

[0047] Preferably, the voxel-level organ tissue and lesion semantic segmentation of the full modality feature map to obtain the corresponding organ tissue semantic map and lesion semantic map specifically includes:

[0048] The organ tissue semantic segmentation processing is performed on the full modality feature map based on a preset organ tissue semantic segmentation model to obtain the corresponding organ tissue semantic map; the organ tissue semantic segmentation model is implemented based on a U-Net network model, a DeepLabv3 model or a DeepLabv3+ model; the three-dimensional size of the organ tissue semantic map is consistent with the three-dimensional size of the full modality feature map, and is composed of a plurality of first semantic map voxel points; each of the first semantic map voxel points corresponds to a first semantic type; the first semantic type includes non-organ tissue voxel points, multiple types of human organ voxel points and multiple types of human tissue voxel points;

[0049] And based on a preset lesion semantic segmentation model, the omnimodal feature map is subjected to lesion semantic segmentation processing to obtain the corresponding lesion semantic map; the lesion semantic segmentation model is implemented based on a U-Net network model, a DeepLabv3 model or a DeepLabv3+ model; the three-dimensional size of the lesion semantic map is consistent with the three-dimensional size of the omnimodal feature map, and is composed of a plurality of second semantic map voxel points; each of the second semantic map voxel points corresponds to a second semantic type; the second semantic type includes non-lesion voxel points and lesion voxel points of multiple types of diseases.

[0050] Preferably, the three-dimensional organ tissue structure modeling is performed according to the full modality feature map, the organ tissue semantic map and the lesion semantic map, and the lesion area positioning mark is performed on the modeling structure, specifically including:

[0051] Performing three-dimensional modeling space planning according to the three-dimensional size of the full-modal feature graph to obtain a corresponding first modeling space;

[0052] And in the first modeling space, various types of human organ voxel points and various types of human tissue voxel points are marked according to the organ tissue semantic map; and voxel points belonging to the same type of human organ or the same type of human tissue are clustered to obtain a corresponding first type of organ voxel point set or a first type of tissue voxel point set; and according to each of the first type of organ voxel point sets or the first type of tissue voxel point sets, corresponding human organ or human tissue three-dimensional structure modeling is performed in the first modeling space to obtain a corresponding human organ structure or human tissue structure;

[0053] And marking the lesion voxel points of various diseases in the first modeling space according to the lesion semantic map; and recording the human organ structure or the human tissue structure containing one or more lesion voxel points in the first modeling space as the corresponding diseased organ / tissue structure; and clustering the voxel points belonging to the same type of disease on each of the diseased organs / tissue structures to obtain a corresponding first-category disease voxel point set, and performing corresponding lesion area three-dimensional structure modeling in the current diseased organ / tissue structure according to each of the first-category disease voxel point sets to obtain a corresponding lesion area structure;

[0054] Based on the three-dimensional modeling display software, all the human organ structures and all the human tissue structures in the first modeling space are visualized, and when the total number of the diseased organs / tissue structures is not zero, all the lesion area structures on each displayed diseased organ / tissue structure are highlighted.

[0055] A second aspect of an embodiment of the present invention provides a device for implementing the processing method for three-dimensional lesion positioning based on multimodal images as described in the first aspect above, the device comprising: a multimodal image receiving module, an image preprocessing module, a single-modal registration module, a single-modal feature recognition module, a single-modal feature fusion module, a full-modal feature fusion module, a semantic segmentation module and a lesion area positioning marking module;

[0056] The multimodal image receiving module is used to receive multiple first modal images; each of the first modal images corresponds to a modality type; the modality types include three-dimensional MRI images, three-dimensional CT images, three-dimensional PET images, two-dimensional Xray images and two-dimensional US images;

[0057] The image preprocessing module is used to perform image denoising and artifact removal processing on each of the first modality images to obtain a corresponding first modality preprocessing image; the three-dimensional or two-dimensional size of each of the first modality preprocessing images is consistent with the three-dimensional or two-dimensional size of the corresponding first modality image;

[0058] The single modality registration module is used to record the first modality preprocessing image that matches the modality type with the preset reference image type as the corresponding reference modality image; and spatially register each of the first modality preprocessing images based on the reference modality image to obtain a corresponding first modality registration image; the reference image type includes a three-dimensional MRI image and a three-dimensional CT image; the three-dimensional or two-dimensional size of each of the first modality registration images is consistent with the three-dimensional or two-dimensional size of the corresponding first modality image;

[0059] The single-modal feature recognition module is used to perform feature extraction processing of the corresponding modality on each of the first modal preprocessing images to obtain a corresponding first modal feature image; the three-dimensional or two-dimensional size of each of the first modal feature images is consistent with the three-dimensional or two-dimensional size of the corresponding first modal image;

[0060] The single-modality feature fusion module is used to perform feature fusion on the first modality feature map and the first modality registration map corresponding to each of the first modality preprocessing maps in a feature channel splicing manner to obtain a corresponding first modality fusion feature map;

[0061] The full modal feature fusion module is used to perform full modal feature fusion on all the first modal fusion feature maps obtained to obtain a corresponding full modal feature map;

[0062] The semantic segmentation module is used to perform voxel-level organ tissue and lesion semantic segmentation on the full-modality feature map to obtain corresponding organ tissue semantic map and lesion semantic map;

[0063] The lesion area positioning marking module is used to perform three-dimensional organ tissue structure modeling based on the full modality feature map, the organ tissue semantic map and the lesion semantic map, and to perform lesion area positioning marking on the modeled structure.

[0064] A third aspect of an embodiment of the present invention provides an electronic device, including: a memory, a processor, and a transceiver;

[0065] The processor is used to be coupled to the memory, read and execute instructions in the memory, so as to implement the method steps described in the first aspect above;

[0066] The transceiver is coupled to the processor, and the processor controls the transceiver to send and receive messages.

[0067] A fourth aspect of an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions. When the computer instructions are executed by a computer, the computer executes the instructions of the method described in the first aspect above.

[0068] The embodiment of the present invention provides a processing method, device, electronic device and computer-readable storage medium for three-dimensional lesion positioning based on multimodal images. As can be seen from the above content, after receiving the multimodal images, the embodiment of the present invention first performs image denoising and artifact removal processing on each modality image respectively, then performs spatial registration and self-modal feature extraction on each modality preprocessing image and fuses the spatial registration feature and self-modal feature to obtain the corresponding single-modal fusion feature map, then performs full-modal feature fusion on all single-modal fusion feature maps to obtain the corresponding full-modal feature map, then performs voxel-level organ tissue and lesion semantic segmentation on the full-modal feature map to obtain the corresponding organ tissue semantic map and lesion semantic map, then performs three-dimensional organ tissue structure modeling based on the full-modal feature map, organ tissue semantic map and lesion semantic map, and performs lesion area positioning marking on the modeling structure. On the one hand, the embodiment of the present invention improves the recognition accuracy of the lesion area through multimodal feature fusion, and on the other hand, improves the recognition accuracy of the lesion area through voxel-level semantic segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 A schematic diagram of a processing method for three-dimensional lesion localization based on multimodal imaging provided in Embodiment 1 of the present invention;

[0070] Figure 2 A module structure diagram of the joint training model provided in the first embodiment of the present invention;

[0071] Figure 3 A module structure diagram of a processing device for three-dimensional lesion localization based on multimodal images provided in Embodiment 2 of the present invention;

[0072] Figure 4 A schematic diagram of the structure of an electronic device provided in Embodiment 3 of the present invention. DETAILED DESCRIPTION

[0073] In order to make the purpose, technical solution and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0074] Embodiment 1 of the present invention provides a processing method for three-dimensional lesion localization based on multimodal imaging, such as Figure 1 As shown in the schematic diagram of a processing method for three-dimensional lesion localization based on multimodal images provided in the first embodiment of the present invention, the method mainly includes the following steps:

[0075] Step 1: receiving a plurality of first modality images.

[0076] Here, each first modality image in the embodiment of the present invention corresponds to a modality type; the modality type in the embodiment of the present invention includes a three-dimensional MRI image, a three-dimensional CT image, a three-dimensional PET image, a two-dimensional Xray image, and a two-dimensional US image;

[0077] The first modality image whose modality type is a three-dimensional MRI image includes a plurality of first voxel points; each first voxel point includes a first voxel coordinate and a first MRI feature vector; each first MRI feature vector is composed of one or more MRI image feature data;

[0078] The first modality image whose modality type is a three-dimensional CT image includes a plurality of second voxel points; each second voxel point includes a second voxel coordinate and a first CT feature vector; each first CT feature vector is composed of one or more CT image feature data;

[0079] The first modality image whose modality type is a three-dimensional PET image includes a plurality of third voxel points; each third voxel point includes a third voxel coordinate and a first PET feature vector; each first PET feature vector is composed of one or more PET image feature data;

[0080] The first modality image whose modality type is a two-dimensional Xray image includes a plurality of first pixel points; each first pixel point includes a first pixel coordinate and a first Xray feature vector; each first Xray feature vector is composed of one or more Xray image feature data;

[0081] The first modality image whose modality type is a two-dimensional US image includes a plurality of second pixel points; each second pixel point includes a second pixel coordinate and a first US feature vector; and each first US feature vector is composed of one or more US image feature data.

[0082] Step 2, performing image denoising and artifact removal processing on each first modality image to obtain a corresponding first modality preprocessing image;

[0083] The three-dimensional or two-dimensional size of each first modality preprocessing image is consistent with the three-dimensional or two-dimensional size of the corresponding first modality image.

[0084] Here, the image denoising and artifact removal processing technologies for MRI / CT / PET / Xray / US images are all conventional image preprocessing technical solutions. The embodiments of the present invention preprocess each first modality image based on these conventional technical solutions and use the processing results as the corresponding preprocessing image, namely the first modality preprocessing image.

[0085] Step 3, recording the first modality preprocessing image whose modality type matches the preset reference image type as the corresponding reference modality image; and performing spatial registration on each first modality preprocessing image based on the reference modality image to obtain a corresponding first modality registration image;

[0086] Specifically, it includes: step 31, recording the first modality preprocessing image whose modality type matches the preset reference image type as the corresponding reference modality image;

[0087] The reference image types include three-dimensional MRI images and three-dimensional CT images, and three-dimensional CT images are generally preferred as the default value of the reference image type;

[0088] Step 32, spatially registering each first modal preprocessing image based on the reference modal image to obtain a corresponding first modal registration image;

[0089] Wherein, the three-dimensional or two-dimensional size of each first modality registration image is consistent with the three-dimensional or two-dimensional size of the corresponding first modality image;

[0090] A first modality registration map corresponding to a first modality image whose modality type is a three-dimensional MRI image, a three-dimensional CT image, or a three-dimensional PET image includes a plurality of fourth voxel points; the fourth voxel points correspond one-to-one to the first, second, or third voxel points of the corresponding first modality image; each fourth voxel point includes a fourth voxel coordinate and a first registration feature vector; each first registration feature vector is composed of a first rotation feature vector and a first displacement feature vector; each first rotation feature vector is composed of matrix parameters of a three-dimensional rotation matrix; each first displacement feature vector is composed of matrix parameters of a three-dimensional displacement matrix;

[0091] A first modality registration map corresponding to a first modality image whose modality type is a two-dimensional Xray image or a two-dimensional US image includes a plurality of third pixel points; the third pixel points correspond one-to-one to the first or second pixel points of the corresponding first modality image; each third pixel point includes a third pixel coordinate and a second registration feature vector; each second registration feature vector is composed of a second rotation feature vector and a second displacement feature vector; each second rotation feature vector is composed of matrix parameters of a two-dimensional rotation matrix; each second displacement feature vector is composed of matrix parameters of a two-dimensional displacement matrix;

[0092] The current step 32 specifically includes:

[0093] Step 3201, taking each first modal preprocessing graph as the corresponding current modal preprocessing graph;

[0094] Step 3202, and taking the modality type corresponding to the current modality preprocessing graph as the corresponding current floating modality type;

[0095] Step 3203, identifying the reference image type and the current floating mode type;

[0096] Step 3204: if the reference image type is the same as the current floating modality type, then construct a first modality registration image whose three-dimensional size is consistent with the three-dimensional size of the current modality preprocessing image; and uniformly set all first rotation feature vectors of the currently constructed first modality registration image based on a preset zero rotation feature vector; and uniformly set all first displacement feature vectors of the currently constructed first modality registration image based on a preset zero displacement feature vector;

[0097] Here, the zero rotation feature vector is composed of a set of preset three-dimensional rotation matrix parameters, and the rotation relationship embodied by the set of preset three-dimensional rotation matrix parameters is a non-rotation relationship, that is, a zero rotation relationship; the zero displacement feature vector is composed of a set of preset three-dimensional displacement matrix parameters, and the displacement relationship embodied by the set of preset three-dimensional displacement matrix parameters is a non-displacement relationship, that is, a zero displacement relationship;

[0098] The conventional three-dimensional rotation matrix is ​​expressed as follows:

[0099]

[0100] Among them, R xyz is a three-dimensional rotation matrix, α, β, θ are the rotation angles around the x, y, and z axes respectively, R x , R y , R z They are the rotation matrices of the x, y, and z axes respectively; in the above three-dimensional rotation matrix R xyz In the expression of , the matrix parameters of the three-dimensional rotation matrix of the embodiment of the present invention are composed of the angle values ​​of the three rotation angles α, β, and θ; then the three-dimensional rotation matrix parameters corresponding to the zero rotation relationship refer to the angle values ​​of the above three rotation angles α, β, and θ being 0;

[0101] In addition, the conventional three-dimensional displacement matrix (also called translation matrix) is expressed as follows:

[0102]

[0103] Among them, T xyz is a three-dimensional displacement matrix, △x, △y, △z are the displacement distances along the x, y, and z axes; in the above three-dimensional displacement matrix T xyz In the expression of , the matrix parameters of the three-dimensional displacement matrix of the embodiment of the present invention are composed of the distance values ​​of the three displacement distances △x, △y, and △z; then the three-dimensional displacement matrix parameters corresponding to the zero displacement relationship refer to the distance values ​​of the above three displacement distances △x, △y, and △z being 0;

[0104] Step 3205: If the reference image type and the current floating modality type are the corresponding three-dimensional MRI image and three-dimensional CT image, respectively, the MRI reference-CT floating registration model in the preset registration model library is used as the corresponding current registration model; and the reference modality image and the current modality preprocessing image are input into the current registration model as the corresponding MRI reference image and CT floating image for configuration processing to obtain the corresponding first modality registration image;

[0105] Step 3206: If the reference image type and the current floating modality type are the corresponding three-dimensional MRI image and three-dimensional PET image, respectively, the MRI reference-PET floating registration model in the registration model library is used as the corresponding current registration model; and the reference modality image and the current modality preprocessing image are input into the current registration model as the corresponding MRI reference image and PET floating image for configuration processing to obtain the corresponding first modality registration image;

[0106] Step 3207: If the reference image type and the current floating modality type are the corresponding three-dimensional MRI image and two-dimensional Xray image, respectively, the MRI reference-Xray floating registration model in the registration model library is used as the corresponding current registration model; and the reference modality image and the current modality preprocessing image are input into the current registration model as the corresponding MRI reference image and Xray floating image for configuration processing to obtain the corresponding first modality registration image;

[0107] Step 3208: If the reference image type and the current floating modality type are the corresponding three-dimensional MRI image and two-dimensional US image, respectively, the MRI reference-US floating registration model in the registration model library is used as the corresponding current registration model; and the reference modality image and the current modality preprocessing image are input into the current registration model as the corresponding MRI reference image and US floating image for configuration processing to obtain the corresponding first modality registration image;

[0108] Step 3209: If the reference image type and the current floating modality type are the corresponding three-dimensional CT image and three-dimensional MRI image, respectively, the CT reference-MRI floating registration model in the preset registration model library is used as the corresponding current registration model; and the reference modality image and the current modality preprocessing image are input into the current registration model as the corresponding CT reference image and MRI floating image for configuration processing to obtain the corresponding first modality registration image;

[0109] Step 3210: If the reference image type and the current floating modality type are the corresponding three-dimensional CT image and three-dimensional PET image, respectively, the CT reference-PET floating registration model in the preset registration model library is used as the corresponding current registration model; and the reference modality image and the current modality preprocessing image are input into the current registration model as the corresponding CT reference image and PET floating image for configuration processing to obtain the corresponding first modality registration image;

[0110] Step 3211: if the reference image type and the current floating modality type are the corresponding three-dimensional CT image and two-dimensional Xray image respectively, the CT reference-Xray floating registration model in the preset registration model library is used as the corresponding current registration model; and the reference modality image and the current modality preprocessing image are input into the current registration model as the corresponding CT reference image and Xray floating image for configuration processing to obtain the corresponding first modality registration image;

[0111] Step 3212: If the reference image type and the current floating modality type are the corresponding three-dimensional CT image and two-dimensional US image, respectively, the CT reference-US floating registration model in the preset registration model library is used as the corresponding current registration model; and the reference modality image and the current modality pre-processing image are input into the current registration model as the corresponding CT reference image and US floating image for configuration processing to obtain the corresponding first modality registration image;

[0112] Step 3213, and output the first modality registration image obtained this time as the spatial registration result of the current modality preprocessing image.

[0113] It should be noted that the registration model library mentioned in the above steps 3205-3212 includes eight types of reference modality-floating modality registration models, namely: MRI reference-CT floating registration model, MRI reference-PET floating registration model, MRI reference-Xray floating registration model, MRI reference-US floating registration model, CT reference-MRI floating registration model, CT reference-PET floating registration model, CT reference-Xray floating registration model and CT reference-US floating registration model; the eight types of reference modality-floating modality registration models in the registration model library of the embodiment of the present invention are each implemented based on a deep learning neural network model.

[0114] The following is a brief description of the model functions of these eight types of reference mode-floating mode registration models.

[0115] 1) MRI benchmark-CT floating registration model:

[0116] The MRI reference-CT floating registration model of the embodiment of the present invention is used to use the MRI reference image and the CT floating image input by the model as the corresponding current reference image and the current floating image; and identify the best matching voxel point of each voxel point of the current floating image on the current reference image according to the similarity measurement relationship between the MRI image features and the CT image features according to the non-rigid registration mechanism; if it is identified that a voxel point on the current floating image has a corresponding best matching voxel point on the current reference image, then the corresponding three-dimensional registration is performed based on the voxel coordinates of the current voxel point on the current floating image and the voxel coordinates of the corresponding best matching voxel point on the current reference image. The first modality registration map is composed of a first rotation matrix parameter and a first displacement matrix parameter. The first registration feature vector and the first displacement feature vector are used as the corresponding first rotation feature vector and the first displacement feature vector to form a corresponding first registration feature vector. If it is identified that a voxel point on the current floating map does not have a corresponding best matching voxel point on the current reference map, the corresponding first registration feature vector is set as a preset first invalid registration feature vector. The voxel coordinates corresponding to each voxel point on the current floating map and the first registration feature vector form a corresponding fourth voxel point, and all the fourth voxel points obtained form a corresponding first modality registration map and output it. Here, the first invalid registration feature vector is a preset invalid vector, and the first rotation feature vector and the first displacement feature vector of the invalid vector are respectively a corresponding invalid three-dimensional rotation matrix parameter group and an invalid three-dimensional displacement matrix parameter group. Each angle value of the invalid three-dimensional rotation matrix parameter group is set to a value beyond the normal angle range, and each distance value of the invalid three-dimensional displacement matrix parameter group is set to a large value beyond the normal displacement range.

[0117] The MRI benchmark-CT floating registration model of the embodiment of the present invention is implemented based on a deep learning neural network model. The model structure of the model consists of two parallel voxel-level image feature extraction networks, a voxel-level image feature similarity comparison network and a voxel-level regression calculation network; wherein, the two voxel-level image feature extraction networks are respectively implemented based on convolutional neural networks (CNN), residual networks (ResNet) or densely connected convolutional networks (DenseNet), the voxel-level image feature similarity comparison network is implemented based on a multilayer perceptron (MLP) neural network, and the voxel-level regression calculation network is implemented based on a fully connected neural network (FCN); the inputs of the two voxel-level image feature extraction networks correspond to the MRI reference image and CT floating image of the model input respectively, and the outputs are connected to the input end of the voxel-level image feature similarity comparison network, the output of the voxel-level image feature similarity comparison network is connected to the input end of the voxel-level regression calculation network, and the output of the voxel-level regression calculation network is connected to the output end of the model. Two parallel voxel-level image feature extraction networks are used for feature extraction of the corresponding MRI reference image and CT floating image respectively, and the extracted voxel-level feature tensors are output to the voxel-level image feature similarity comparison network. The voxel-level image feature similarity comparison network is used to compare the image feature similarity according to the two input voxel-level feature tensors, and identify the voxel coordinates and corresponding similarities of the reference image voxel points in the MRI reference image that are most similar to each floating image voxel point on the CT floating image, and form a corresponding similarity comparison vector from the obtained reference image voxel point coordinates and reference image voxel point similarity, and form a corresponding voxel-level similarity tensor from the similarity comparison vectors of all floating image voxel points to output to the voxel-level regression calculation network.The voxel-level regression calculation network is used to record the similarity comparison vectors whose similarity of the reference image voxel points in the voxel-level similarity tensor is lower than the similarity threshold as invalid vectors, and record the similarity comparison vectors whose similarity is not lower than the similarity threshold as valid vectors according to a preset similarity threshold, and set a corresponding first registration feature vector for each invalid vector based on the preset first invalid registration feature vector, and confirm the corresponding three-dimensional rotation matrix parameters and three-dimensional displacement matrix parameters based on the reference image voxel point coordinates of each valid vector and the floating image voxel point coordinates corresponding to the current valid vector, and use the confirmed two sets of matrix parameters as the corresponding first rotation feature vector and the first displacement feature vector to form a corresponding first registration feature vector, and use the voxel coordinates corresponding to each floating voxel point on the current floating image and the first registration feature vector to form a corresponding fourth voxel point, and all the fourth voxel points obtained form a corresponding first modal registration map and output it.

[0118] The MRI benchmark-CT floating registration model of the embodiment of the present invention needs to be trained based on a supervised training method before the model is used. The data set used for training consists of multiple data records, each of which consists of a pair of MRI benchmark images and CT floating images and a label modality registration image generated by manual annotation or other machine annotation methods. In the model training process, the data set is split into two data record subsets as corresponding training sets and evaluation sets according to the preset total number of records of the training set and the evaluation set, and the MRI benchmark-CT floating registration model is trained and the model parameters are optimized based on the training set and the preset model training loss function, and the MRI benchmark-CT floating registration model is evaluated based on the evaluation set and the preset model evaluation function. The model training loss function here defaults to L1 loss function or L2 loss function, and other loss functions can also be selected according to the actual application requirements for setting, and the model evaluation function defaults to mean square error (MSE) function or root mean square error (RMSE) function, and other error functions can also be selected according to the actual application requirements for setting.

[0119] 2) MRI benchmark-PET floating registration model:

[0120] The MRI reference-PET floating registration model of the embodiment of the present invention is used to use the MRI reference image and the PET floating image input by the model as the corresponding current reference image and the current floating image; and identify the best matching voxel point of each voxel point of the current floating image on the current reference image according to the similarity measurement relationship between the MRI image features and the PET image features according to the non-rigid registration mechanism; if it is identified that a voxel point on the current floating image has a corresponding best matching voxel point on the current reference image, then the voxel coordinates of the current voxel point on the current floating image are compared with the voxel coordinates of the corresponding best matching voxel point on the current reference image. The corresponding three-dimensional rotation matrix parameters and three-dimensional displacement matrix parameters are confirmed, and the two sets of matrix parameters are used as the corresponding first rotation feature vector and the first displacement feature vector to form a corresponding first registration feature vector; if it is identified that a voxel point on the current floating image does not have a corresponding best matching voxel point on the current reference image, the corresponding first registration feature vector is set to a first invalid registration feature vector; and the voxel coordinates corresponding to each voxel point on the current floating image and the first registration feature vector form a corresponding fourth voxel point, and all the fourth voxel points obtained form a corresponding first modal registration map and output it.

[0121] The MRI benchmark-PET floating registration model of the embodiment of the present invention is implemented based on a deep learning neural network model. The model structure of the MRI benchmark-PET floating registration model of the embodiment of the present invention is similar to the model structure of the MRI benchmark-CT floating registration model and is composed of two parallel voxel-level image feature extraction networks, a voxel-level image feature similarity comparison network and a voxel-level regression calculation network.

[0122] The network functions of each sub-network of the MRI benchmark-PET floating registration model of the embodiment of the present invention are similar to the corresponding sub-network functions in the MRI benchmark-CT floating registration model, and will not be further described here.

[0123] The MRI reference-PET floating registration model of the embodiment of the present invention also needs to be trained based on a supervised training method before the model is used. The data set used for training consists of multiple data records, each of which consists of a pair of MRI reference images and PET floating images and a label modality registration image generated by manual annotation or other machine annotation methods. During the training process, the training / evaluation set splitting method of the data set, the selection of the model training loss function and the model evaluation function, the training and parameter optimization method of the model based on the training set and the model training loss function, and the performance evaluation method of the model based on the evaluation set and the model evaluation function are similar to the processing method of the MRI reference-CT floating registration model, and will not be further explained here.

[0124] 3) MRI benchmark-Xray floating registration model:

[0125] The MRI reference-Xray floating registration model of the embodiment of the present invention is used to use the MRI reference image and the Xray floating image input by the model as the corresponding current reference image and the current floating image; and perform a two-dimensional plane projection on the current reference image to obtain the corresponding current reference projection image; and identify the best matching pixel point of each pixel point of the current floating image on the current reference projection image according to the similarity measurement relationship between the MRI image features and the Xray image features according to the non-rigid registration mechanism; if it is identified that a certain pixel point on the current floating image has a corresponding best matching pixel point on the current reference image, then based on the pixel coordinates of each pixel point on the current floating image and the corresponding pixel point on the current reference projection image, The pixel coordinates of the best matching pixel point are used to confirm the corresponding two-dimensional rotation matrix parameters and two-dimensional displacement matrix parameters, and the two sets of matrix parameters are used as the corresponding second rotation feature vector and the second displacement feature vector to form a corresponding second registration feature vector; if it is identified that a certain pixel point on the current floating image does not have a corresponding best matching pixel point on the current reference image, the corresponding second registration feature vector is set to a preset second invalid registration feature vector; and the pixel coordinates corresponding to each pixel point on the current floating image and the second registration feature vector form a corresponding third pixel point, and all the obtained third pixel points form a corresponding first modality registration map and output it.

[0126] The MRI reference-Xray floating registration model of the embodiment of the present invention is implemented based on a deep learning neural network model. The model structure of the model consists of two parallel pixel-level image feature extraction networks, a pixel-level image feature similarity comparison network and a pixel-level regression calculation network; wherein the two pixel-level image feature extraction networks are respectively implemented based on a CNN network, a ResNet network or a DenseNet network, the pixel-level image feature similarity comparison network is implemented based on an MLP neural network, and the voxel-level regression calculation network is implemented based on an FCN network; the inputs of the two pixel-level image feature extraction networks correspond to the MRI reference image and the Xray floating image of the model input respectively, and the outputs are connected to the input end of the pixel-level image feature similarity comparison network, the output of the pixel-level image feature similarity comparison network is connected to the input end of the pixel-level regression calculation network, and the output of the pixel-level regression calculation network is connected to the output end of the model.

[0127] The pixel-level image feature extraction network connected to the MRI reference image is used to perform a two-dimensional plane projection on the MRI reference image to obtain the corresponding current reference projection image and extract features from the current reference projection image to obtain a corresponding pixel-level feature tensor, which is then output to the pixel-level image feature similarity comparison network; the pixel-level image feature extraction network connected to the Xray floating image is used to extract features from the Xray floating image to obtain a corresponding pixel-level feature tensor, which is then output to the pixel-level image feature similarity comparison network. The pixel-level image feature similarity comparison network is used to perform image feature similarity comparison based on two input pixel-level feature tensors, and to identify the pixel coordinates and corresponding similarities of the reference projection image pixel points in the current reference projection image that are most similar to each floating image pixel point on the Xray floating image, and to form a corresponding similarity comparison vector from the obtained reference projection image pixel coordinates and reference projection image pixel point similarities, and to form a corresponding pixel-level similarity tensor from the similarity comparison vectors of all floating image pixel points, which is then output to the pixel-level regression calculation network. The pixel regression calculation network is used to record the similarity comparison vectors whose similarity of the pixels of the reference projection image in the pixel-level similarity tensor is lower than the similarity threshold as invalid vectors and the similarity comparison vectors whose similarity is not lower than the similarity threshold as valid vectors according to a preset similarity threshold, and set a corresponding second registration feature vector for each invalid vector based on the preset second invalid registration feature vector, and confirm the corresponding two-dimensional rotation matrix parameters and two-dimensional displacement matrix parameters based on the pixel coordinates of the reference projection image of each valid vector and the pixel coordinates of the floating image corresponding to the current valid vector, and use the confirmed two sets of matrix parameters as the corresponding second rotation feature vector and the second displacement feature vector to form a corresponding second registration feature vector, and use the pixel coordinates and the second registration feature vector corresponding to each floating image pixel on the current floating image to form a corresponding third pixel point, and use all the obtained third pixels to form a corresponding first modality registration map and output it.

[0128] The MRI benchmark-Xray floating registration model of the embodiment of the present invention needs to be trained based on a supervised training method before the model is used. The data set used for training consists of multiple data records, each of which consists of a pair of MRI benchmark images and Xray floating images and a label modality registration image generated by manual annotation or other machine annotation methods. In the model training process, the data set is split into two data record subsets as corresponding training sets and evaluation sets according to the preset ratio of the total number of records of the training set and the evaluation set, and the MRI benchmark-Xray floating registration model is trained and the model parameters are optimized based on the training set and the preset model training loss function, and the MRI benchmark-Xray floating registration model is evaluated based on the evaluation set and the preset model evaluation function. The model training loss function here defaults to L1 loss function or L2 loss function, and other loss functions can also be selected according to the actual application requirements for setting, and the model evaluation function defaults to MSE function or RMSE function, and other error functions can also be selected according to the actual application requirements for setting.

[0129] 4) MRI benchmark-US floating registration model:

[0130] The MRI reference-US floating registration model of the embodiment of the present invention is used to use the MRI reference image and the US floating image input by the model as the corresponding current reference image and the current floating image; and perform a two-dimensional plane projection on the current reference image to obtain the corresponding current reference projection image; and identify the best matching pixel point of each pixel point of the current floating image on the current reference projection image according to the similarity measurement relationship between the MRI image features and the US image features according to the non-rigid registration mechanism; if it is identified that a certain pixel point on the current floating image has a corresponding best matching pixel point on the current reference image, then based on the pixel coordinates of each pixel point on the current floating image and the corresponding best matching pixel point on the current reference projection image, The pixel coordinates of the best matching pixel point are used to confirm the corresponding two-dimensional rotation matrix parameters and two-dimensional displacement matrix parameters, and the two sets of matrix parameters are used as the corresponding second rotation feature vector and the second displacement feature vector to form a corresponding second registration feature vector; if it is identified that a certain pixel point on the current floating image does not have a corresponding best matching pixel point on the current reference image, the corresponding second registration feature vector is set to a second invalid registration feature vector; and the pixel coordinates corresponding to each pixel point on the current floating image and the second registration feature vector are used to form a corresponding third pixel point, and all the obtained third pixel points form a corresponding first modality registration map and output it.

[0131] The MRI benchmark-US floating registration model of the embodiment of the present invention is implemented based on a deep learning neural network model. The model structure of the MRI benchmark-US floating registration model of the embodiment of the present invention is similar to the model structure of the MRI benchmark-Xray floating registration model and is composed of two parallel pixel-level image feature extraction networks, a pixel-level image feature similarity comparison network and a pixel-level regression calculation network.

[0132] The network functions of each sub-network of the MRI benchmark-US floating registration model in the embodiment of the present invention are similar to the corresponding sub-network functions in the MRI benchmark-Xray floating registration model, and will not be further described here.

[0133] The MRI benchmark-US floating registration model of the embodiment of the present invention also needs to be trained based on a supervised training method before the model is used. The data set used for training consists of multiple data records, each of which consists of a pair of MRI benchmark images and US floating images and a label modality registration image generated by manual annotation or other machine annotation methods. During the training process, the training / evaluation set splitting method of the data set, the selection of the model training loss function and the model evaluation function, the training and parameter optimization method of the model based on the training set and the model training loss function, and the performance evaluation method of the model based on the evaluation set and the model evaluation function are similar to the processing method of the MRI benchmark-Xray floating registration model, and will not be further explained here.

[0134] 5) CT benchmark-MRI floating registration model:

[0135] The CT reference-MRI floating registration model of the embodiment of the present invention is used to use the CT reference image and the MRI floating image input by the model as the corresponding current reference image and the current floating image; and identify the best matching voxel point of each voxel point of the current floating image on the current reference image according to the similarity measurement relationship between the CT image feature and the MRI image feature according to the non-rigid registration mechanism; if it is identified that a voxel point on the current floating image has a corresponding best matching voxel point on the current reference image, then the voxel coordinates of the current voxel point on the current floating image are matched with the voxel coordinates of the corresponding best matching voxel point on the current reference image. The three-dimensional rotation matrix parameters and the three-dimensional displacement matrix parameters are confirmed, and the two sets of matrix parameters are used as the corresponding first rotation feature vector and the first displacement feature vector to form a corresponding first registration feature vector; if it is identified that a voxel point on the current floating image does not have a corresponding best matching voxel point on the current reference image, the corresponding first registration feature vector is set to a first invalid registration feature vector; and the voxel coordinates corresponding to each voxel point on the current floating image and the first registration feature vector form a corresponding fourth voxel point, and all the fourth voxel points obtained form a corresponding first modal registration map and output it.

[0136] The CT benchmark-MRI floating registration model of the embodiment of the present invention is implemented based on a deep learning neural network model. The model structure of the CT benchmark-MRI floating registration model of the embodiment of the present invention is similar to the model structure of the MRI benchmark-CT floating registration model and is composed of two parallel voxel-level image feature extraction networks, a voxel-level image feature similarity comparison network and a voxel-level regression calculation network.

[0137] The network functions of each sub-network of the CT benchmark-MRI floating registration model in the embodiment of the present invention are similar to the corresponding sub-network functions in the MRI benchmark-CT floating registration model, and will not be further described here.

[0138] The CT benchmark-MRI floating registration model of the embodiment of the present invention also needs to be trained based on a supervised training method before the model is used. The data set used for training consists of multiple data records, each of which consists of a pair of CT benchmark images and MRI floating images and a label modality registration image generated by manual annotation or other machine annotation methods. During the training process, the training / evaluation set splitting method of the data set, the selection of the model training loss function and the model evaluation function, the training and parameter optimization method of the model based on the training set and the model training loss function, and the performance evaluation method of the model based on the evaluation set and the model evaluation function are similar to the processing method of the MRI benchmark-CT floating registration model, and will not be further explained here.

[0139] 6) CT benchmark-PET floating registration model:

[0140] The CT reference-PET floating registration model of the embodiment of the present invention is used to use the CT reference image and the PET floating image input by the model as the corresponding current reference image and the current floating image; and identify the best matching voxel point of each voxel point of the current floating image on the current reference image according to the similarity measurement relationship between the CT image feature and the PET image feature according to the non-rigid registration mechanism; if it is identified that a voxel point on the current floating image has a corresponding best matching voxel point on the current reference image, then the voxel coordinates of the current voxel point on the current floating image are matched with the voxel coordinates of the corresponding best matching voxel point on the current reference image. The three-dimensional rotation matrix parameters and the three-dimensional displacement matrix parameters are confirmed, and the two sets of matrix parameters are used as the corresponding first rotation feature vector and the first displacement feature vector to form a corresponding first registration feature vector; if it is identified that a voxel point on the current floating image does not have a corresponding best matching voxel point on the current reference image, the corresponding first registration feature vector is set to a first invalid registration feature vector; and the voxel coordinates corresponding to each voxel point on the current floating image and the first registration feature vector form a corresponding fourth voxel point, and all the fourth voxel points obtained form a corresponding first modal registration map and output it.

[0141] The CT benchmark-PET floating registration model of the embodiment of the present invention is implemented based on a deep learning neural network model. The model structure of the CT benchmark-PET floating registration model of the embodiment of the present invention is similar to the model structure of the MRI benchmark-CT floating registration model and is composed of two parallel voxel-level image feature extraction networks, a voxel-level image feature similarity comparison network and a voxel-level regression calculation network.

[0142] The network functions of each sub-network of the CT benchmark-PET floating registration model in the embodiment of the present invention are similar to the corresponding sub-network functions in the MRI benchmark-CT floating registration model, and will not be further described here.

[0143] The CT reference-PET floating registration model of the embodiment of the present invention also needs to be trained based on a supervised training method before the model is used. The data set used for training consists of multiple data records, each of which consists of a pair of CT reference images and PET floating images and a label modality registration image generated by manual annotation or other machine annotation methods. During the training process, the training / evaluation set splitting method of the data set, the selection of the model training loss function and the model evaluation function, the training and parameter optimization method of the model based on the training set and the model training loss function, and the performance evaluation method of the model based on the evaluation set and the model evaluation function are similar to the processing method of the MRI reference-CT floating registration model, and will not be further explained here.

[0144] 7) CT benchmark-Xray floating registration model:

[0145] The CT reference-Xray floating registration model of the embodiment of the present invention is used to use the CT reference image and the Xray floating image input by the model as the corresponding current reference image and the current floating image; and perform a two-dimensional plane projection on the current reference image to obtain the corresponding current reference projection image; and identify the best matching pixel point of each pixel point of the current floating image on the current reference projection image according to the similarity measurement relationship between the CT image feature and the Xray image feature according to the non-rigid registration mechanism; if it is identified that a certain pixel point on the current floating image has a corresponding best matching pixel point on the current reference image, then based on the pixel coordinates of each pixel point on the current floating image and the corresponding pixel point on the current reference projection image, The pixel coordinates of the best matching pixel point are used to confirm the corresponding two-dimensional rotation matrix parameters and two-dimensional displacement matrix parameters, and the two sets of matrix parameters are used as the corresponding second rotation feature vector and the second displacement feature vector to form a corresponding second registration feature vector; if it is identified that a certain pixel point on the current floating image does not have a corresponding best matching pixel point on the current reference image, the corresponding second registration feature vector is set to a second invalid registration feature vector; and the pixel coordinates corresponding to each pixel point on the current floating image and the second registration feature vector form a corresponding third pixel point, and all the obtained third pixel points form a corresponding first modality registration map and output it.

[0146] The CT benchmark-Xray floating registration model of the embodiment of the present invention is implemented based on a deep learning neural network model. The model structure of the CT benchmark-Xray floating registration model of the embodiment of the present invention is similar to the model structure of the MRI benchmark-Xray floating registration model and is composed of two parallel pixel-level image feature extraction networks, a pixel-level image feature similarity comparison network and a pixel-level regression calculation network.

[0147] The network functions of each sub-network of the CT benchmark-Xray floating registration model in the embodiment of the present invention are similar to the corresponding sub-network functions in the MRI benchmark-Xray floating registration model, and will not be further described here.

[0148] The CT benchmark-Xray floating registration model of the embodiment of the present invention also needs to be trained based on a supervised training method before the model is used. The data set used for training consists of multiple data records, each of which consists of a pair of CT benchmark images and Xray floating images and a label modality registration image generated by manual annotation or other machine annotation methods. During the training process, the training / evaluation set splitting method of the data set, the selection of the model training loss function and the model evaluation function, the training and parameter optimization method of the model based on the training set and the model training loss function, and the performance evaluation method of the model based on the evaluation set and the model evaluation function are similar to the processing method of the MRI benchmark-Xray floating registration model, and will not be further explained here.

[0149] 8) CT benchmark-US floating registration model:

[0150] The CT reference-US floating registration model of the embodiment of the present invention is used to use the CT reference image and the US floating image input by the model as the corresponding current reference image and the current floating image; and perform a two-dimensional plane projection on the current reference image to obtain the corresponding current reference projection image; and identify the best matching pixel point of each pixel point of the current floating image on the current reference projection image according to the similarity measurement relationship between the CT image feature and the US image feature according to the non-rigid registration mechanism; if it is identified that a certain pixel point on the current floating image has a corresponding best matching pixel point on the current reference image, then based on the pixel coordinates of each pixel point on the current floating image and the corresponding best matching pixel point on the current reference projection image, the pixel coordinates of each pixel point on the current floating image are compared with the pixel coordinates of the pixel point on the current floating image. The pixel coordinates of the matching pixel points are used to confirm the corresponding two-dimensional rotation matrix parameters and two-dimensional displacement matrix parameters, and the two confirmed sets of matrix parameters are used as the corresponding second rotation feature vector and the second displacement feature vector to form a corresponding second registration feature vector; if it is identified that a certain pixel point on the current floating image does not have a corresponding best matching pixel point on the current reference image, the corresponding second registration feature vector is set to a second invalid registration feature vector; and the pixel coordinates and the second registration feature vector corresponding to each pixel point on the current floating image form a corresponding third pixel point, and all the obtained third pixel points form a corresponding first modality registration map and output it.

[0151] The CT benchmark-US floating registration model of the embodiment of the present invention is implemented based on a deep learning neural network model. The model structure of the CT benchmark-US floating registration model of the embodiment of the present invention is similar to the model structure of the MRI benchmark-Xray floating registration model and is composed of two parallel pixel-level image feature extraction networks, a pixel-level image feature similarity comparison network and a pixel-level regression calculation network.

[0152] The network functions of each sub-network of the CT benchmark-US floating registration model in the embodiment of the present invention are similar to the corresponding sub-network functions in the MRI benchmark-Xray floating registration model, and will not be further described here.

[0153] The CT benchmark-US floating registration model of the embodiment of the present invention also needs to be trained based on a supervised training method before the model is used. The data set used for training consists of multiple data records, each of which consists of a pair of CT benchmark images and US floating images and a label modality registration image generated by manual annotation or other machine annotation methods. During the training process, the training / evaluation set splitting method of the data set, the selection of the model training loss function and the model evaluation function, the training and parameter optimization method of the model based on the training set and the model training loss function, and the performance evaluation method of the model based on the evaluation set and the model evaluation function are similar to the processing method of the MRI benchmark-Xray floating registration model, and will not be further explained here.

[0154] Step 4, performing feature extraction processing of the corresponding mode on each first modal preprocessing image to obtain a corresponding first modal feature image;

[0155] Wherein, the three-dimensional or two-dimensional size of each first modality feature map is consistent with the three-dimensional or two-dimensional size of the corresponding first modality image;

[0156] A first modality feature map corresponding to a first modality image whose modality type is a three-dimensional MRI image, a three-dimensional CT image, or a three-dimensional PET image includes a plurality of fifth voxel points; the fifth voxel points correspond one-to-one to the first, second, or third voxel points of the corresponding first modality image; the fifth voxel points include fifth voxel coordinates and a first image feature vector;

[0157] The first modality feature map corresponding to the first modality image whose modality type is a two-dimensional Xray image or a two-dimensional US image includes a plurality of fourth pixel points; the fourth pixel points correspond one-to-one to the first or second pixel points of the corresponding first modality image; the fourth pixel points include a fourth pixel coordinate and a second image feature vector;

[0158] The current step 4 specifically includes:

[0159] Step 41, taking each first modal preprocessing graph as the corresponding current modal preprocessing graph;

[0160] Step 42, and taking the modality type corresponding to the current modality preprocessing image as the corresponding current modality type;

[0161] Step 43, and identify the current modality type; if the current modality type is a three-dimensional MRI image, the MRI image feature extraction model in the preset feature extraction model library is used as the corresponding current feature extraction model; if the current modality type is a three-dimensional CT image, the CT image feature extraction model in the feature extraction model library is used as the corresponding current feature extraction model; if the current modality type is a three-dimensional PET image, the PET image feature extraction model in the feature extraction model library is used as the corresponding current feature extraction model; if the current modality type is a two-dimensional Xray image, the Xray image feature extraction model in the feature extraction model library is used as the corresponding current feature extraction model; if the current modality type is a two-dimensional US image, the US image feature extraction model in the feature extraction model library is used as the corresponding current feature extraction model;

[0162] Step 44, performing feature extraction processing on the current modal preprocessing image based on the current feature extraction model to obtain a corresponding first modal feature image.

[0163] It should be noted that the feature extraction model library mentioned in the above step 43 includes five types of feature extraction models, namely: MRI image feature extraction model, CT image feature extraction model, PET image feature extraction model, Xray image feature extraction model and US image feature extraction model; each type of feature extraction model in the feature extraction model library of the embodiment of the present invention is implemented based on a type of CNN network model, ResNet network model or DenseNet network model.

[0164] The five types of feature extraction models in the feature extraction model library of the embodiment of the present invention also need to be trained based on a supervised training method before the model is used, but they need to be jointly trained with the full-modal coding model, organ tissue semantic segmentation model and lesion semantic segmentation model mentioned below.

[0165] Step 5: Perform feature fusion on the first modality feature maps and the first modality registration maps corresponding to each first modality preprocessing map in a feature channel splicing manner to obtain the corresponding first modality fusion feature map.

[0166] Here, the first modality fusion feature map obtained in the embodiment of the present invention is divided into two categories, one corresponding to the first modality image whose modality type is a three-dimensional MRI image, a three-dimensional CT image, or a three-dimensional PET image, and the other corresponding to the first modality image whose modality type is a two-dimensional Xray image or a two-dimensional US image;

[0167] Wherein, the first modality fusion feature map corresponding to the first modality image whose modality type is a three-dimensional MRI image, a three-dimensional CT image or a three-dimensional PET image includes a plurality of sixth voxel points; the sixth voxel points correspond one-to-one to the first, second or third voxel points of the corresponding first modality image; the sixth voxel points include the sixth voxel coordinates and the first fusion feature vector; the first fusion feature vector is composed of the corresponding first image feature vector and the first registration feature vector;

[0168] A first modality fusion feature map corresponding to a first modality image whose modality type is a two-dimensional Xray image or a two-dimensional US image includes multiple fifth pixel points; the fifth pixel points correspond one-to-one to the first or second pixel points of the corresponding first modality image; the fifth pixel points include fifth pixel coordinates and a second fusion feature vector; and the second fusion feature vector is composed of the corresponding second image feature vector and the second registration feature vector.

[0169] Step 6, performing full-modal feature fusion on all the first-modal fusion feature maps obtained to obtain corresponding full-modal feature maps;

[0170] Specifically, the method includes: taking the first modality fusion feature map corresponding to the reference image type as the corresponding reference modality fusion feature map; and planning a three-dimensional space as the corresponding fusion coding space based on the three-dimensional size of the reference modality fusion feature map; and using a preset full-modality coding model to perform full-modality feature fusion coding processing in the fusion coding space according to all first modality fusion feature maps to obtain the corresponding full-modality feature map;

[0171] Among them, the omnimodal coding model is implemented based on a type of multimodal autoencoder (AE) model or a type of multimodal Transformer model.

[0172] Here, the omnimodal coding model of an embodiment of the present invention also needs to be trained based on a supervised training method before the model is used, but it needs to be jointly trained with the five types of feature extraction models of the feature extraction model library mentioned above and the organ tissue semantic segmentation model and lesion semantic segmentation model mentioned below.

[0173] Step 7, performing voxel-level organ tissue and lesion semantic segmentation on the full modality feature map to obtain the corresponding organ tissue semantic map and lesion semantic map;

[0174] Specifically, the method comprises: step 71, performing organ tissue semantic segmentation processing on the full modality feature map based on a preset organ tissue semantic segmentation model to obtain a corresponding organ tissue semantic map;

[0175] The organ tissue semantic segmentation model is implemented based on the U-Net network model, the DeepLabv3 model or the DeepLabv3+ model; the three-dimensional size of the organ tissue semantic map is consistent with the three-dimensional size of the full-modal feature map, and is composed of multiple first semantic map voxel points; each first semantic map voxel point corresponds to a first semantic type; the first semantic type includes non-organ tissue voxel points, multiple types of human organ voxel points and multiple types of human tissue voxel points;

[0176] Step 72, performing lesion semantic segmentation processing on the full modality feature map based on a preset lesion semantic segmentation model to obtain a corresponding lesion semantic map;

[0177] Among them, the lesion semantic segmentation model is implemented based on the U-Net network model, DeepLabv3 model or DeepLabv3+ model; the three-dimensional size of the lesion semantic map is consistent with the three-dimensional size of the full-modal feature map, and is composed of multiple second semantic map voxel points; each second semantic map voxel point corresponds to a second semantic type; the second semantic type includes non-lesion voxel points and lesion voxel points of multiple types of diseases.

[0178] Here, the organ tissue semantic segmentation model and the lesion semantic segmentation model of the embodiment of the present invention also need to be trained based on a supervised training method before the model is used, but they need to be jointly trained with the five types of feature extraction models and the omnimodal coding model of the feature extraction model library mentioned above. The training method of this joint training is briefly described below.

[0179] 1) Build a joint training model:

[0180] A joint training model is composed of five feature extraction models (MRI image feature extraction model, CT image feature extraction model, PET image feature extraction model, Xray image feature extraction model and US image feature extraction model), a full-modality coding model, an organ tissue semantic segmentation model and a lesion semantic segmentation model. Figure 2 This is a module structure diagram of the joint training model provided in Example 1 of the present invention.

[0181] 2) Constructing a joint dataset:

[0182] Collect multiple sets of multimodal images to construct corresponding joint data sets, including:

[0183] Step A1, acquiring multiple multimodal image sequences;

[0184] Here, each multimodal image sequence includes at least one MRI image and one CT image, and may also include one PET image and / or one Xray image and / or one US image;

[0185] Step A2, recording the images in each multimodal image sequence that match the preset reference image type as the corresponding reference modality map, and performing spatial registration on each image in the current multimodal image sequence based on the reference modality map to obtain the corresponding MRI registration map, CT registration map and / or PET registration map and / or Xray registration map and / or US registration map;

[0186] Step A3, using medical simulation software to perform high-precision simulation of the three-dimensional structure of real human organs and tissues according to the MRI images and CT images of each multimodal image sequence to obtain a corresponding first simulation structure; and referring to the preset human organ and tissue type set based on manual annotation or other machine annotation methods, voxel-level semantic annotation of various human organs and voxel-level semantic annotation of various human tissues are performed on the first simulation structure, and a corresponding first label semantic map is generated based on the result; and according to the real disease type and real lesion location of the corresponding person in the current multimodal image sequence, voxel-level semantic annotation of disease lesions is performed on the first simulation structure through manual annotation or other machine annotation methods, and a corresponding second label semantic map is generated based on the result;

[0187] Step A4, the MRI image + MRI registration map, CT image + CT registration map and / or PET image + PET registration map and / or Xray image + Xray registration map and / or US image + US registration map corresponding to each multimodal image sequence form a corresponding first data sequence; and the first and second label semantic maps corresponding to each multimodal image sequence form a corresponding first label sequence; and the first data sequence and the first label sequence corresponding to each multimodal image sequence form a corresponding joint data record;

[0188] Step A5: All the obtained joint data records form a corresponding joint data set.

[0189] 3) Construct the first training set and the first evaluation set:

[0190] The joint data set is split into two data subsets as a corresponding first training set and a first evaluation set according to a preset first ratio. Here, the first ratio is a preset ratio value, for example, 8:2. If the first ratio is 8:2, it means that 80% of the joint data records are randomly selected from the joint data set to form a first training set, and the remaining 20% ​​of the joint data records form a first evaluation set.

[0191] 4) Construct model loss function L total and the model evaluation function C total :

[0192] Set a corresponding model loss function L for the joint training model total and a corresponding model evaluation function C total ;in,

[0193] Model loss function L total With the model evaluation function C total As shown below:

[0194] L total =w a L a (y a,i ,y′ a,i )+w b L b (y b,i ,y′ b,i ),

[0195]

[0196] w a 、w b are two preset weight parameters; L a is the preset first semantic segmentation loss function, L b is the preset second semantic segmentation loss function. The first and second semantic segmentation loss functions default to multi-classification cross entropy loss functions, and other loss functions can also be selected for setting based on actual application requirements; i is the record index of the first training set, 1≤i≤M, and M is the total number of joint data records of the first training set; y a,i ,y' a,i are the first predicted semantic graph and the first label semantic graph corresponding to the i-th joint data record of the first training set, respectively, b,i ,y' b,i are respectively a second predicted semantic map and a second label semantic map corresponding to the i-th joint data record of the first training set;

[0197] q a ,q b are two preset weight parameters respectively; j is the record index of the first evaluation set, 1≤j≤N, N is the total number of joint data records of the first evaluation set; y a,j ,y' a,j are the first predicted semantic graph and the first label semantic graph corresponding to the i-th joint data record in the first evaluation set, respectively, b,j ,y' b,j are the second predicted semantic graph and the second label semantic graph corresponding to the i-th joint data record in the first evaluation set, respectively.

[0198] 5) Train the joint training model:

[0199] Based on the first training set and model loss function L total Perform a round of joint training on the joint training model, including:

[0200] Step B1, set the record index i to 1; and count the total number of joint data records in the first training set and use the statistical result as the corresponding total number of joint data records M;

[0201] Step B2, extracting the i-th joint data record of the first training set as the corresponding current data record;

[0202] Step B3, inputting the MRI / CT / PET / Xray / US image in the first data sequence of the current data record into the corresponding MRI / CT / PET / Xray / US image feature extraction model in the joint training model to perform feature extraction to obtain the corresponding MRI / CT / PET / Xray / US feature map;

[0203] Step B4, inputting the obtained MRI / CT / PET / Xray / US feature map and the corresponding MRI / CT / PET / Xray / US registration map in the first data sequence of the current data record into the corresponding MRI / CT / PET / Xray / US feature splicing module in the joint training model, and each feature splicing module performs feature fusion on the input feature map and registration map in a feature channel splicing manner to obtain the corresponding MRI / CT / PET / Xray / US fusion feature map;

[0204] Step B5, inputting all obtained fusion feature maps into the full-modal coding model of the joint training model to perform full-modal feature fusion coding processing to obtain the corresponding first full-modal feature map;

[0205] Step B6: input the first full-modal feature map into the organ tissue semantic segmentation model and the lesion semantic segmentation model of the joint training model respectively to perform corresponding organ tissue semantic segmentation and lesion semantic segmentation to obtain the corresponding first predicted semantic map y a,i and the second predicted semantic graph y b,i ;

[0206] Step B7: record the first and second label semantic graphs of the first label sequence of the current data record as the corresponding first label semantic graph y' a,i and the second label semantic graph y' b,i ; and the first predicted semantic graph y a,i , the second predicted semantic graph y b,i , the first label semantic graph y' a,i , the second label semantic graph y' b,i Substitute the model loss function L total Calculate and obtain the corresponding first loss value;

[0207] Step B8, identify whether the first loss value meets the preset first loss value range; if the first loss value meets the first loss value range, identify whether the record index i is less than the total number of joint data records M, if so, add 1 to the record index i and return to step B2, if not, go to step B9; if the first loss value does not meet the first loss value range, then based on the preset model parameter optimizer, move towards making the model loss function L total A round of joint modulation optimization is performed on the model parameters of the five-category feature extraction model, the omnimodal coding model, the organ tissue semantic segmentation model, and the lesion semantic segmentation model of the joint training model in the direction of reaching the minimum value, and at the end of this round of optimization, the model returns to step B3 to continue training;

[0208] Here, the first loss value range is a preset loss value range, and the model parameter optimizer includes at least an SDG optimizer;

[0209] Step B9, solidify the model parameters of the five types of feature extraction models, the omnimodal encoding model, the organ tissue semantic segmentation model and the lesion semantic segmentation model in the joint training model, and confirm the end of this round of joint training.

[0210] 6) Evaluate the joint training model:

[0211] Based on the first evaluation set and the model evaluation function C total Evaluate the joint training model, including:

[0212] Step C1, counting the total number of joint data records in the first evaluation set and using the counting result as the corresponding total number N of joint data records;

[0213] Step C2: input the first data sequence of each joint data record of the first evaluation set into the joint training model for model prediction to obtain the corresponding first prediction semantic graph y a,j , the second predicted semantic graph y b,j ; and the first and second label semantic graphs of the first label sequence of each joint data record of the first evaluation set are recorded as the corresponding first label semantic graph y' a,j and the second label semantic graph y' b,j ;

[0214] Step C3: obtain the N groups of first predicted semantic graphs y a,j , the second predicted semantic graph y b,j , the first label semantic graph y' a,j and the second label semantic graph y' b,j Substitute into the model evaluation function C total Calculating to obtain a corresponding first evaluation value;

[0215] Step C4, identifying whether the first evaluation value meets the preset first evaluation value range; if not, confirming that the training maturity of the joint training model does not meet the standard, and giving the first training set and the model loss function L total A prompt message is given to conduct another round of joint training on the joint training model; if satisfied, it is confirmed that the training maturity of the joint training model has reached the standard, and a prompt message is given that the five-category feature extraction model, the omnimodal coding model, the organ tissue semantic segmentation model and the lesion semantic segmentation model have completed the model training.

[0216] Step 8, performing three-dimensional organ tissue structure modeling based on the full modality feature map, organ tissue semantic map and lesion semantic map, and marking the lesion area on the modeled structure;

[0217] Specifically, the method comprises: step 81, performing three-dimensional modeling space planning according to the three-dimensional size of the full-modal feature graph to obtain a corresponding first modeling space;

[0218] Step 82, in the first modeling space, various types of human organ voxel points and various types of human tissue voxel points are marked according to the organ-tissue semantic map; and voxel points belonging to the same type of human organ or the same type of human tissue are clustered to obtain a corresponding first type of organ voxel point set or a first type of tissue voxel point set; and corresponding human organ or human tissue three-dimensional structure modeling is performed in the first modeling space according to each first type of organ voxel point set or first type of tissue voxel point set to obtain a corresponding human organ structure or human tissue structure;

[0219] Step 83, marking the lesion voxel points of various diseases in the first modeling space according to the lesion semantic map; recording the human organ structure or human tissue structure containing one or more lesion voxel points in the first modeling space as the corresponding diseased organ / tissue structure; clustering the voxel points belonging to the same type of disease on each diseased organ / tissue structure to obtain a corresponding first-category disease voxel point set, and performing corresponding lesion area three-dimensional structure modeling in the current diseased organ / tissue structure according to each first-category disease voxel point set to obtain a corresponding lesion area structure;

[0220] Step 84, based on the three-dimensional modeling display software, all human organ structures and all human tissue structures in the first modeling space are visualized, and when the total number of diseased organs / tissue structures is not zero, all lesion area structures on each displayed diseased organ / tissue structure are highlighted.

[0221] Figure 3The module structure diagram of a processing device for three-dimensional lesion localization based on multimodal imaging provided in the second embodiment of the present invention, the device is a terminal device or server that implements the aforementioned method embodiment, and can also be a device that enables the aforementioned terminal device or server to implement the aforementioned method embodiment, for example, the device can be a device or chip system of the aforementioned terminal device or server. Figure 3 As shown, the device includes: a multimodal image receiving module 201, an image preprocessing module 202, a single-modal registration module 203, a single-modal feature recognition module 204, a single-modal feature fusion module 205, an all-modal feature fusion module 206, a semantic segmentation module 207 and a lesion area positioning marking module 208.

[0222] The multimodal image receiving module 201 is used to receive multiple first modal images; each first modal image corresponds to a modality type; the modality types include three-dimensional MRI images, three-dimensional CT images, three-dimensional PET images, two-dimensional Xray images and two-dimensional US images.

[0223] The image preprocessing module 202 is used to perform image denoising and artifact removal processing on each first modality image to obtain a corresponding first modality preprocessing image; the three-dimensional or two-dimensional size of each first modality preprocessing image is consistent with the three-dimensional or two-dimensional size of the corresponding first modality image.

[0224] The single modality registration module 203 is used to record the first modality preprocessing image whose modality type matches the preset reference image type as the corresponding reference modality image; and to spatially register each first modality preprocessing image based on the reference modality image to obtain a corresponding first modality registration image; the reference image types include three-dimensional MRI images and three-dimensional CT images; the three-dimensional or two-dimensional size of each first modality registration image is consistent with the three-dimensional or two-dimensional size of the corresponding first modality image.

[0225] The single-modal feature recognition module 204 is used to perform feature extraction processing of the corresponding modality on each first-modal preprocessing image to obtain a corresponding first-modal feature image; the three-dimensional or two-dimensional size of each first-modal feature image is consistent with the three-dimensional or two-dimensional size of the corresponding first-modal image.

[0226] The single modality feature fusion module 205 is used to perform feature fusion on the first modality feature map and the first modality registration map corresponding to each first modality preprocessing map in a feature channel splicing manner to obtain a corresponding first modality fusion feature map.

[0227] The full modal feature fusion module 206 is used to perform full modal feature fusion on all the obtained first modal fusion feature maps to obtain corresponding full modal feature maps.

[0228] The semantic segmentation module 207 is used to perform voxel-level organ tissue and lesion semantic segmentation on the full-modality feature map to obtain corresponding organ tissue semantic map and lesion semantic map.

[0229] The lesion area positioning marking module 208 is used to perform three-dimensional organ tissue structure modeling based on the full modality feature map, the organ tissue semantic map and the lesion semantic map, and to perform lesion area positioning marking on the modeled structure.

[0230] An embodiment of the present invention provides a processing device for three-dimensional lesion localization based on multimodal images, which can execute the method steps in the above method embodiment. Its implementation principle and technical effects are similar and will not be repeated here.

[0231] It should be noted that it should be understood that the division of the various modules of the above device is only a division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or they can be physically separated. And these modules can all be implemented in the form of software called by processing elements; they can also be all implemented in the form of hardware; some modules can also be implemented in the form of software called by processing elements, and some modules can be implemented in the form of hardware. For example, the multimodal image receiving module can be a separately established processing element, or it can be integrated in a chip of the above device. In addition, it can also be stored in the memory of the above device in the form of program code, and called and executed by a processing element of the above device. The function of the above-determined module. The implementation of other modules is similar. In addition, these modules can be fully or partially integrated together, or they can be implemented independently. The processing element described here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each module above can be completed by an integrated logic circuit of hardware in the processor element or instructions in the form of software.

[0232] For example, the above modules may be one or more integrated circuits configured to implement the above methods, such as one or more application specific integrated circuits (ASIC), or one or more digital signal processors (DSP), or one or more field programmable gate arrays (FPGA). For another example, when a module above is implemented in the form of a processing element scheduling program code, the processing element may be a general-purpose processor, such as a central processing unit (CPU) or other processor that can call program code. For another example, these modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0233] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the above method embodiments are generated. The above computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The above-mentioned computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the above-mentioned computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, Bluetooth, microwave, etc.) methods. The above-mentioned computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media integrated. The above-mentioned available medium can be a magnetic medium (such as a floppy disk, a hard disk, a tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid state disk (SSD)), etc.

[0234] Figure 4 This is a schematic diagram of the structure of an electronic device provided in Embodiment 3 of the present invention. The electronic device may be a terminal device or a server for implementing the method of the aforementioned embodiment, or may be a terminal device or a server for implementing the method of the aforementioned embodiment connected to the aforementioned terminal device or server. Figure 4 As shown, the electronic device may include: a processor 301 (such as a CPU), a memory 302, and a transceiver 303; the transceiver 303 is coupled to the processor 301, and the processor 301 controls the transceiver 303. Various instructions may be stored in the memory 302 to complete various processing functions and implement the processing steps described in the aforementioned embodiment method. Preferably, the electronic device involved in the embodiment of the present invention also includes: a power supply 304, a system bus 305 and a communication port 306. The system bus 305 is used to realize the communication connection between components. The above-mentioned communication port 306 is used for connecting and communicating between the electronic device and other peripherals.

[0235] exist Figure 4The system bus 305 mentioned in the figure can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The system bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. The communication interface is used to realize the communication between the database access device and other devices (such as clients, read-write libraries, and read-only libraries). The memory may include random access memory (RAM), and may also include non-volatile memory (Non-Volatile Memory), such as at least one disk storage.

[0236] The above-mentioned processor can be a general-purpose processor, including a central processing unit CPU, a network processor (Network Processor, NP), a graphics processing unit (Graphics Processing Unit, GPU), etc.; it can also be a digital signal processor DSP, an application-specific integrated circuit ASIC, a field programmable gate array FPGA or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0237] It should be noted that an embodiment of the present invention further provides a computer-readable storage medium, in which instructions are stored. When the computer-readable storage medium is run on a computer, the computer executes the method and processing process provided in the above embodiments.

[0238] An embodiment of the present invention further provides a chip for executing instructions, wherein the chip is used to execute the processing steps described in the aforementioned method embodiment.

[0239] The embodiment of the present invention provides a processing method, device, electronic device and computer-readable storage medium for three-dimensional lesion positioning based on multimodal images. As can be seen from the above content, after receiving the multimodal images, the embodiment of the present invention first performs image denoising and artifact removal processing on each modality image respectively, then performs spatial registration and self-modal feature extraction on each modality preprocessing image and fuses the spatial registration feature and self-modal feature to obtain the corresponding single-modal fusion feature map, then performs full-modal feature fusion on all single-modal fusion feature maps to obtain the corresponding full-modal feature map, then performs voxel-level organ tissue and lesion semantic segmentation on the full-modal feature map to obtain the corresponding organ tissue semantic map and lesion semantic map, and then performs three-dimensional organ tissue structure modeling based on the full-modal feature map, organ tissue semantic map and lesion semantic map and performs lesion area positioning marking on the modeling structure. On the one hand, the embodiment of the present invention improves the recognition accuracy of the lesion area through multimodal feature fusion, and on the other hand, improves the recognition accuracy of the lesion area through voxel-level semantic segmentation.

[0240] The professionals should further realize that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to the function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0241] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0242] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A processing method for three-dimensional lesion localization based on multimodal imaging, characterized in that: The method comprises: Receiving a plurality of first modality images; each of the first modality images corresponds to a modality type; the modality types include three-dimensional MRI images, three-dimensional CT images, three-dimensional PET images, two-dimensional X-ray images, and two-dimensional US images; Performing image denoising and artifact removal processing on each of the first modality images to obtain a corresponding first modality preprocessing image; the three-dimensional or two-dimensional size of each of the first modality preprocessing images is consistent with the three-dimensional or two-dimensional size of the corresponding first modality image; The first modality preprocessing image whose modality type matches the preset reference image type is recorded as a corresponding reference modality image; and each of the first modality preprocessing images is spatially registered based on the reference modality image to obtain a corresponding first modality registration image; the reference image type includes a three-dimensional MRI image and a three-dimensional CT image; the three-dimensional or two-dimensional size of each of the first modality registration images is consistent with the three-dimensional or two-dimensional size of the corresponding first modality image; Performing feature extraction processing of the corresponding modality on each of the first modality preprocessing images to obtain a corresponding first modality feature image; the three-dimensional or two-dimensional size of each of the first modality feature images is consistent with the three-dimensional or two-dimensional size of the corresponding first modality image; Performing feature fusion on the first modality feature map and the first modality registration map corresponding to each of the first modality preprocessing maps in a feature channel splicing manner to obtain a corresponding first modality fusion feature map; Performing full-modal feature fusion on all the first modal fusion feature maps obtained to obtain corresponding full-modal feature maps; Performing voxel-level organ tissue and lesion semantic segmentation on the full-modality feature map to obtain corresponding organ tissue semantic map and lesion semantic map; Performing three-dimensional organ tissue structure modeling according to the full modality feature map, the organ tissue semantic map and the lesion semantic map, and marking the lesion area location on the modeled structure; The step of performing full-modal feature fusion on all the first modal fusion feature maps to obtain a corresponding full-modal feature map specifically includes: The first modal fusion feature map corresponding to the reference image type is used as the corresponding reference modal fusion feature map; and a three-dimensional space is planned as the corresponding fusion coding space based on the three-dimensional size of the reference modal fusion feature map; and a preset full modal coding model is used to perform full modal feature fusion coding processing in the fusion coding space according to all the first modal fusion feature maps to obtain the corresponding full modal feature map; the full modal coding model is implemented based on a type of multimodal autoencoder model or a type of multimodal Transformer model.

2. The method for three-dimensional lesion localization based on multimodal imaging according to claim 1, characterized in that: The first modality image whose modality type is a three-dimensional MRI image includes a plurality of first voxel points; the first voxel points include first voxel coordinates and a first MRI feature vector; the first MRI feature vector is composed of one or more MRI image feature data; The first modality image whose modality type is a three-dimensional CT image includes a plurality of second voxel points; the second voxel points include second voxel coordinates and a first CT feature vector; the first CT feature vector is composed of one or more CT image feature data; The first modality image whose modality type is a three-dimensional PET image includes a plurality of third voxel points; the third voxel points include third voxel coordinates and a first PET feature vector; the first PET feature vector is composed of one or more PET image feature data; The first modality image whose modality type is a two-dimensional Xray image includes a plurality of first pixel points; the first pixel points include first pixel coordinates and a first Xray feature vector; the first Xray feature vector is composed of one or more Xray image feature data; The first modality image whose modality type is a two-dimensional US image includes a plurality of second pixel points; the second pixel points include second pixel coordinates and a first US feature vector; the first US feature vector is composed of one or more US image feature data; The first modality registration map corresponding to the first modality image whose modality type is a three-dimensional MRI image, a three-dimensional CT image or a three-dimensional PET image includes a plurality of fourth voxel points; the fourth voxel points correspond one-to-one to the first, second or third voxel points of the corresponding first modality image; the fourth voxel points include fourth voxel coordinates and a first registration feature vector; the first registration feature vector is composed of a first rotation feature vector and a first displacement feature vector; the first rotation feature vector is composed of matrix parameters of a three-dimensional rotation matrix; the first displacement feature vector is composed of matrix parameters of a three-dimensional displacement matrix; The first modality feature map corresponding to the first modality image whose modality type is a three-dimensional MRI image, a three-dimensional CT image, or a three-dimensional PET image includes a plurality of fifth voxel points; the fifth voxel points correspond one-to-one to the first, second, or third voxel points of the corresponding first modality image; the fifth voxel points include fifth voxel coordinates and a first image feature vector; The first modality registration map corresponding to the first modality image whose modality type is a two-dimensional Xray image or a two-dimensional US image includes a plurality of third pixel points; the third pixel points correspond one-to-one to the first or second pixel points of the corresponding first modality image; the third pixel points include third pixel coordinates and a second registration feature vector; the second registration feature vector is composed of a second rotation feature vector and a second displacement feature vector; the second rotation feature vector is composed of matrix parameters of a two-dimensional rotation matrix; the second displacement feature vector is composed of matrix parameters of a two-dimensional displacement matrix; The first modality feature map corresponding to the first modality image whose modality type is a two-dimensional Xray image or a two-dimensional US image includes a plurality of fourth pixel points; the fourth pixel points correspond one-to-one to the first or second pixel points of the corresponding first modality image; the fourth pixel points include fourth pixel coordinates and a second image feature vector; The first modality fusion feature map corresponding to the first modality image whose modality type is a three-dimensional MRI image, a three-dimensional CT image, or a three-dimensional PET image includes a plurality of sixth voxel points; the sixth voxel points correspond one-to-one to the first, second, or third voxel points of the corresponding first modality image; the sixth voxel points include sixth voxel coordinates and a first fusion feature vector; the first fusion feature vector is composed of the corresponding first image feature vector and the first registration feature vector; The first modality fusion feature map corresponding to the first modality image whose modality type is a two-dimensional Xray image or a two-dimensional US image includes multiple fifth pixel points; the fifth pixel points correspond one-to-one to the first or second pixel points of the corresponding first modality image; the fifth pixel points include fifth pixel coordinates and a second fusion feature vector; the second fusion feature vector is composed of the corresponding second image feature vector and the second registration feature vector.

3. The method for three-dimensional lesion localization based on multimodal imaging according to claim 2, characterized in that: The spatially registering each of the first modal preprocessing images based on the reference modal image to obtain a corresponding first modal registration image specifically includes: Using each of the first modal preprocessing graphs as the corresponding current modal preprocessing graph; and taking the modality type corresponding to the current modality preprocessing graph as the corresponding current floating modality type; and identifying the reference image type and the current floating modality type; If the reference image type is the same as the current floating modality type, constructing a first modality registration image whose three-dimensional size is consistent with the three-dimensional size of the current modality preprocessing image; and uniformly setting all the first rotation feature vectors of the first modality registration image currently constructed based on a preset zero rotation feature vector; and uniformly setting all the first displacement feature vectors of the first modality registration image currently constructed based on a preset zero displacement feature vector; If the reference image type and the current floating modality type are the corresponding three-dimensional MRI image and three-dimensional CT image, respectively, the MRI reference-CT floating registration model in the preset registration model library is used as the corresponding current registration model; and the reference modality image and the current modality preprocessing image are input into the current registration model as the corresponding MRI reference image and CT floating image for configuration processing to obtain the corresponding first modality registration image; If the reference image type and the current floating modality type are the corresponding three-dimensional MRI image and three-dimensional PET image respectively, the MRI reference-PET floating registration model in the registration model library is used as the corresponding current registration model; and the reference modality image and the current modality pre-processing image are input into the current registration model as the corresponding MRI reference image and PET floating image for configuration processing to obtain the corresponding first modality registration image; If the reference image type and the current floating modality type are corresponding three-dimensional MRI images and two-dimensional Xray images, respectively, the MRI reference-Xray floating registration model in the registration model library is used as the corresponding current registration model; and the reference modality image and the current modality preprocessing image are input into the current registration model as the corresponding MRI reference image and Xray floating image for configuration processing to obtain the corresponding first modality registration image; If the reference image type and the current floating modality type are corresponding three-dimensional MRI images and two-dimensional US images, respectively, the MRI reference-US floating registration model in the registration model library is used as the corresponding current registration model; and the reference modality image and the current modality pre-processing image are input into the current registration model as the corresponding MRI reference image and US floating image for configuration processing to obtain the corresponding first modality registration image; If the reference image type and the current floating modality type are corresponding three-dimensional CT images and three-dimensional MRI images, respectively, a CT reference-MRI floating registration model in a preset registration model library is used as the corresponding current registration model; and the reference modality image and the current modality preprocessing image are input into the current registration model as the corresponding CT reference image and MRI floating image for configuration processing to obtain the corresponding first modality registration image; If the reference image type and the current floating modality type are corresponding three-dimensional CT images and three-dimensional PET images, respectively, a CT reference-PET floating registration model in a preset registration model library is used as the corresponding current registration model; and the reference modality image and the current modality preprocessing image are input into the current registration model as the corresponding CT reference image and PET floating image for configuration processing to obtain the corresponding first modality registration image; If the reference image type and the current floating modality type are corresponding three-dimensional CT images and two-dimensional Xray images, respectively, the CT reference-Xray floating registration model in the preset registration model library is used as the corresponding current registration model; and the reference modality map and the current modality preprocessing map are input into the current registration model as the corresponding CT reference map and Xray floating map for configuration processing to obtain the corresponding first modality registration map; If the reference image type and the current floating modality type are corresponding three-dimensional CT images and two-dimensional US images, respectively, a CT reference-US floating registration model in a preset registration model library is used as the corresponding current registration model; and the reference modality image and the current modality preprocessing image are input into the current registration model as the corresponding CT reference image and US floating image for configuration processing to obtain the corresponding first modality registration image; And the first modality registration image obtained this time is output as the spatial registration result of the current modality preprocessing image.

4. The method for three-dimensional lesion localization based on multimodal imaging according to claim 3, characterized in that: The registration model library includes the MRI benchmark-CT floating registration model, the MRI benchmark-PET floating registration model, the MRI benchmark-Xray floating registration model, the MRI benchmark-US floating registration model, the CT benchmark-MRI floating registration model, the CT benchmark-PET floating registration model, the CT benchmark-Xray floating registration model and the CT benchmark-US floating registration model; each registration model in the registration model library is implemented based on a deep learning neural network model.

5. The method for three-dimensional lesion localization based on multimodal imaging according to claim 2, characterized in that: The step of performing feature extraction processing of the corresponding modality on each of the first modal preprocessing images to obtain the corresponding first modal feature image specifically includes: Using each of the first modal preprocessing graphs as the corresponding current modal preprocessing graph; and taking the modality type corresponding to the current modality preprocessing graph as the corresponding current modality type; And identify the current modality type; if the current modality type is a three-dimensional MRI image, the MRI image feature extraction model in the preset feature extraction model library is used as the corresponding current feature extraction model; if the current modality type is a three-dimensional CT image, the CT image feature extraction model in the feature extraction model library is used as the corresponding current feature extraction model; if the current modality type is a three-dimensional PET image, the PET image feature extraction model in the feature extraction model library is used as the corresponding current feature extraction model; if the current modality type is a two-dimensional Xray image, the Xray image feature extraction model in the feature extraction model library is used as the corresponding current feature extraction model; if the current modality type is a two-dimensional US image, the US image feature extraction model in the feature extraction model library is used as the corresponding current feature extraction model; And based on the current feature extraction model, feature extraction processing is performed on the current modal preprocessing image to obtain the corresponding first modal feature image.

6. The method for three-dimensional lesion localization based on multimodal imaging according to claim 5, characterized in that: The feature extraction model library includes the MRI image feature extraction model, the CT image feature extraction model, the PET image feature extraction model, the Xray image feature extraction model and the US image feature extraction model; Each feature extraction model of the feature extraction model library is implemented based on a type of convolutional neural network model, a type of residual network model or a type of densely connected convolutional network model.

7. The method for three-dimensional lesion localization based on multimodal imaging according to claim 1, characterized in that: The voxel-level organ tissue and lesion semantic segmentation of the full modality feature map to obtain the corresponding organ tissue semantic map and lesion semantic map specifically includes: The organ tissue semantic segmentation processing is performed on the full modality feature map based on a preset organ tissue semantic segmentation model to obtain the corresponding organ tissue semantic map; the organ tissue semantic segmentation model is implemented based on a U-Net network model, a DeepLabv3 model or a DeepLabv3+ model; the three-dimensional size of the organ tissue semantic map is consistent with the three-dimensional size of the full modality feature map, and is composed of a plurality of first semantic map voxel points; each of the first semantic map voxel points corresponds to a first semantic type; the first semantic type includes non-organ tissue voxel points, multiple types of human organ voxel points and multiple types of human tissue voxel points; And based on a preset lesion semantic segmentation model, the omnimodal feature map is subjected to lesion semantic segmentation processing to obtain the corresponding lesion semantic map; the lesion semantic segmentation model is implemented based on a U-Net network model, a DeepLabv3 model or a DeepLabv3+ model; the three-dimensional size of the lesion semantic map is consistent with the three-dimensional size of the omnimodal feature map, and is composed of a plurality of second semantic map voxel points; each of the second semantic map voxel points corresponds to a second semantic type; the second semantic type includes non-lesion voxel points and lesion voxel points of multiple types of diseases.

8. The method for three-dimensional lesion localization based on multimodal imaging according to claim 1, characterized in that: The three-dimensional organ tissue structure modeling according to the full modality feature map, the organ tissue semantic map and the lesion semantic map and the lesion area positioning marking on the modeling structure specifically includes: Performing three-dimensional modeling space planning according to the three-dimensional size of the full-modal feature graph to obtain a corresponding first modeling space; And in the first modeling space, various types of human organ voxel points and various types of human tissue voxel points are marked according to the organ tissue semantic map; and voxel points belonging to the same type of human organ or the same type of human tissue are clustered to obtain a corresponding first type of organ voxel point set or a first type of tissue voxel point set; and according to each of the first type of organ voxel point sets or the first type of tissue voxel point sets, corresponding human organ or human tissue three-dimensional structure modeling is performed in the first modeling space to obtain a corresponding human organ structure or human tissue structure; And marking the lesion voxel points of various diseases in the first modeling space according to the lesion semantic map; and recording the human organ structure or the human tissue structure containing one or more lesion voxel points in the first modeling space as the corresponding diseased organ / tissue structure; and clustering the voxel points belonging to the same type of disease on each of the diseased organs / tissue structures to obtain a corresponding first-category disease voxel point set, and performing corresponding lesion area three-dimensional structure modeling in the current diseased organ / tissue structure according to each of the first-category disease voxel point sets to obtain a corresponding lesion area structure; Based on the three-dimensional modeling display software, all the human organ structures and all the human tissue structures in the first modeling space are visualized, and when the total number of the diseased organs / tissue structures is not zero, all the lesion area structures on each displayed diseased organ / tissue structure are highlighted.

9. A device for executing the processing method for three-dimensional lesion localization based on multimodal images as described in any one of claims 1 to 8, characterized in that: The device comprises: a multimodal image receiving module, an image preprocessing module, a single-modal registration module, a single-modal feature recognition module, a single-modal feature fusion module, an all-modal feature fusion module, a semantic segmentation module and a lesion area positioning marking module; The multimodal image receiving module is used to receive multiple first modal images; each of the first modal images corresponds to a modality type; the modality types include three-dimensional MRI images, three-dimensional CT images, three-dimensional PET images, two-dimensional Xray images and two-dimensional US images; The image preprocessing module is used to perform image denoising and artifact removal processing on each of the first modality images to obtain a corresponding first modality preprocessing image; the three-dimensional or two-dimensional size of each of the first modality preprocessing images is consistent with the three-dimensional or two-dimensional size of the corresponding first modality image; The single modality registration module is used to record the first modality preprocessing image that matches the modality type with the preset reference image type as the corresponding reference modality image; and spatially register each of the first modality preprocessing images based on the reference modality image to obtain a corresponding first modality registration image; the reference image type includes a three-dimensional MRI image and a three-dimensional CT image; the three-dimensional or two-dimensional size of each of the first modality registration images is consistent with the three-dimensional or two-dimensional size of the corresponding first modality image; The single-modal feature recognition module is used to perform feature extraction processing of the corresponding modality on each of the first modal preprocessing images to obtain a corresponding first modal feature image; the three-dimensional or two-dimensional size of each of the first modal feature images is consistent with the three-dimensional or two-dimensional size of the corresponding first modal image; The single-modality feature fusion module is used to perform feature fusion on the first modality feature map and the first modality registration map corresponding to each of the first modality preprocessing maps in a feature channel splicing manner to obtain a corresponding first modality fusion feature map; The full modal feature fusion module is used to perform full modal feature fusion on all the first modal fusion feature maps obtained to obtain a corresponding full modal feature map; The semantic segmentation module is used to perform voxel-level organ tissue and lesion semantic segmentation on the full-modality feature map to obtain corresponding organ tissue semantic map and lesion semantic map; The lesion area positioning marking module is used to perform three-dimensional organ tissue structure modeling based on the full modality feature map, the organ tissue semantic map and the lesion semantic map, and to perform lesion area positioning marking on the modeled structure.

10. An electronic device, characterized in that: include: memory, processors, and transceivers; The processor is used to couple with the memory, read and execute instructions in the memory, so as to implement the method according to any one of claims 1 to 8; The transceiver is coupled to the processor, and the processor controls the transceiver to send and receive messages.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and when the computer instructions are executed by a computer, the computer is enabled to execute the method according to any one of claims 1 to 8.

Images