Multi-organ localization method in multimodal medical images based on one-to-one target query Transformer

By using the combination of conditional Gaussian model and Transformer network in multimodal medical images, a position correlation model between organs is constructed, which solves the problem of limited organ positioning performance in multimodal medical images, and achieves a fast and accurate multi-organ Improved automatic positioning and learning convergence speed.

CN114359642BActive Publication Date: 2024-11-19DALIAN UNIV OF TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210030228.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-01-12
Publication Date
2024-11-19
Estimated Expiration
2042-01-12

AI Technical Summary

Technical Problem

The prior art lacks understanding of the geometric relationships between organs in images when localizing organs in multimodal medical images, resulting in limited positioning performance. The Transformer model is trained for a long time and requires a large amount of labeled data, and does not use multimodal information. Improve accuracy.

Method used

Multi-modal medical image multi-organ positioning method based on one-to-one target query Transformer, through the combination of conditional Gaussian model and Transformer network, the position correlation between the multi-modal image is constructed using two-dimensional projection and grayscale normalization of multi-modal images. and organ positioning through self-attention mechanisms, simplifying network structure and reducing redundant calculations.

Benefits of technology

Fast and accurate automatic positioning of multi-organs is achieved, network structure is simplified, computing burden is reduced, learning convergence speed is improved, and positioning accuracy is enhanced using multimodal information.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114359642B_ABST
    Figure CN114359642B_ABST
Patent Text Reader

Abstract

The present invention provides a multi-organ localization method for multi-modal medical images based on a one-to-one target query Transformer, and belongs to the technical field of medical image processing. The present invention utilizes a conditional Gaussian model and the self-attention mechanism of the Transformer to simulate the correlation between the position and size of organs. The one-to-one target query architecture enforces a unique target query for each target organ, and the order of the query is the predicted category, so there is no need for classification, which simplifies the network structure, reduces redundant calculations, and has a faster learning convergence speed. Before performing organ detection, the 3D multimodal image is projected onto two orthogonal 2D planes, and then the complementary information from the multimodal image is combined through a multimodal fusion method. Finally, the obtained 2D bounding box is back-projected to obtain a 3D bounding box, which reduces the computational burden and obtains a more robust organ localization result.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of medical image processing, and specifically relates to a multi-modal medical image multi-organ localization method based on a one-to-one target query Transformer, which mainly utilizes the geometric correlation between organs and the information complementarity of multi-modal medical images to efficiently and accurately perform automatic organ localization. Background Art

[0002] Multimodal medical images are common imaging modes in clinical medicine, such as positron emission computed tomography / computed tomography (PET / CT), T1, T2, and proton density maps in nuclear magnetic resonance scanning, and multiple spectral images in spectral CT. Taking PET / CT images as an example, PET images are functional images that reflect the distribution of radioactive tracers in the body and can be used for diagnosis of benign and malignant tumors and quantification of tissue metabolism; CT images reflect the degree of absorption of X-rays by human organs and tissues and can clearly display the anatomical structure of the human body. With the popularization of multimodal medical imaging, a large amount of image data is obtained every day, and the diagnostic burden of doctors is increasing. Automatic organ localization methods can help reduce image reading time and provide specific organ areas for subsequent computer-aided diagnosis. Therefore, fast and accurate automatic localization of multiple organs has become an indispensable step in multimodal medical image analysis.

[0003] Organ localization in an image refers to determining the three-dimensional bounding box of the organ, that is, determining the upper and lower limits of the x, y, and z coordinates of the organ (i.e., x min , x max ,y min ,y max , z min , z max ).

[0004] In recent years, deep convolutional neural networks (CNNs) have performed well in object detection in natural or medical images. Although CNNs have been applied to organ localization in images, their performance is still limited due to the lack of understanding of the geometric relationship between organs in images.

[0005] Recently, the development of Transformer networks has shown the potential of using long-range target correlations for image classification or segmentation, and has gradually been used to model the correlation between targets, thereby improving the performance of medical image analysis. However, the Transformer network has a complex structure and requires a lot of training time; it also requires a large amount of training data to perform well, but it is a time-consuming and laborious task for doctors to manually label a large number of three-dimensional medical images; existing three-dimensional Transformers only maintain a balance between accuracy and learnability by reducing the depth of the network. In addition, current Transformer models do not use information from multimodal medical images to improve the accuracy of organ localization. Summary of the invention

[0006] In order to solve the above problems, the present invention proposes a multi-organ localization method for multi-modal medical images based on a one-to-one target query transformer. The method fuses the two-dimensional projection views (coronal and sagittal directions) of the three-dimensional multi-modal image together for organ detection, uses the conditional Gaussian model (CGM) to model the correlation between organs in the human body region, and extracts features from the multi-modal fusion image using CNN. The obtained image feature sequence and CGM prediction results are used as the input of the transformer. Under the comprehensive effect of the transformer, the one-to-one target query finally obtains the position of multiple organs, the order of the target query corresponds to the category of multiple organs, and the detection results of the two projection views are back-projected into the 3D space to obtain the 3D bounding box, thereby completing the automatic localization of multiple organs.

[0007] The technical solution of the present invention is:

[0008] A multi-modal medical image multi-organ localization method based on a one-to-one target query Transformer comprises the following steps:

[0009] S1. 2D projection and image fusion of 3D multimodal images

[0010] S11. Data preprocessing

[0011] The dataset is preprocessed by normalizing the spatial resolution and image size as follows:

[0012] First, the multimodal images are resampled to a uniform voxel size based on the finest spatial resolution among the multiple modal images. Then, the image size is selected and the resampled images are center cropped to remove surrounding background pixels while retaining the human body area.

[0013] S12. Two-dimensional projection and grayscale normalization of multimodal images

[0014] S121. 3D medical images require a lot of computation, and their projection Figure 1 The main information in the three-dimensional medical image can be characterized to a certain extent, thereby greatly reducing the amount of calculation and reducing the difficulty of positioning. Therefore, each modality image is first projected using a projection method that maintains the image characteristics to obtain a two-dimensional coronal projection and sagittal projection. The details are as follows:

[0015] For functional modality images, such as PET images, organs and tissues with high metabolism are highlighted in the images. In order to highlight organs and tissues with high metabolism, Maximum Intensity Projection (MIP) is selected to ensure good contrast of organs with high metabolism.

[0016] For anatomical modality images, such as CT and MRI images, the anatomical structure can be displayed, and the human eye can observe most of the clearly displayed organ tissues. If MIP is used, high-contrast tissues will be highlighted and low-contrast tissues will be weakened. Therefore, Average Intensity Projection (AIP) is used to maintain tissue contrast.

[0017] S122. For each modality projection image, an appropriate threshold range is selected to normalize the grayscale of the image to the grayscale range of [0,255]. The threshold range specifically uses a normalization window of [g(p1), g(p2)], where g(p1) and g(p2) represent grayscale thresholds that exclude pixels in the image that are lower than the p1 percentage and higher than the p2 percentage.

[0018] S13. Image fusion of multimodal projection

[0019] Each projection view is fused together to generate two fused images of the coronal and sagittal views respectively; the fused image calculation formula is:

[0020]

[0021] Among them, α u represents the weight factor, and I u Represents the projection view of the u-th modality image.

[0022] S2. Building a rough prediction model of position correlation between organs through CGM

[0023] S21. Constructing the shape vector of the organ

[0024] Detection of anatomical structure positions in multimodal images is a key prerequisite for automatic image diagnosis. For organ localization, it is not necessary to accurately mark the organs with contour lines, but only to determine the upper and lower coordinate limits of the three-dimensional bounding box of the organ in the three-dimensional multimodal image.

[0025] Each training sample in the training set has 6 feature values, namely the coordinates of the center point and the length, width and height of the three-dimensional bounding box of the organ, which constitute the training set. For each training sample, a shape vector is constructed, assuming s i is the shape vector of training sample i, i.e. s i =(x, y, z, l x , l y , l z ).

[0026] S22, Generalized Platts Transform

[0027] Before using shape vectors that characterize the position and size of organs for statistical modeling, in order to eliminate other factors, the shape vector of each training sample in the image space needs to be normalized to the model space through the generalized Procrustes transformation, and then subsequent operations are performed in the model space.

[0028] S23. Principal Component Analysis (PCA)

[0029] PCA is derived from linear algebra. Through linear transformation, multiple variables in the data set are sorted according to the weight of the principal components, and some of them can represent the characteristics of the entire data set, thereby reducing the dimension of the data set. When analyzing complex data, it can reduce the amount of calculation, retain the main information, and remove redundant parts and noise. It should be noted that PCA is performed in the model space.

[0030] All s after generalized Protozoa transformation i As the training set, the statistical shape model method is used to model the position and size changes of the organs, and the shape model is expressed as follows:

[0031]

[0032] in, is the average shape of the training set, which is the average value of the features of the selected training set; Φ s is the eigenvector matrix (subscript s indicates shape) obtained by principal component analysis (PCA) of the training set, where the k eigenvectors represent the k organ position and size change patterns learned from the training set; b s is the shape parameter, b s The k elements in Φ represent sThe weight of the k deformation modes in the training set superimposed with the average shape; s represents the final shape model, b s Affect s by adjusting b s The value can control the deformation of the model.

[0033] According to formula (2), the shape parameter of each training sample is inversely calculated as follows:

[0034]

[0035] in, is the transpose of the eigenvector matrix obtained after PCA on the training set, s i is the shape vector of the i-th training sample, is the average shape of the training set, b s is the shape parameter of the training samples.

[0036] S24. Constructing a conditional Gaussian model

[0037] The organs of the human body are highly correlated in position and shape, so the correlation between organs can be modeled using the conditional Gaussian model. and Represent the shape parameters of two adjacent organs A and B respectively. When it is known hour, The conditional Gaussian model CGM description is shown in formulas (4)-(6):

[0038]

[0039]

[0040]

[0041] in, is the mean value of the shape parameters of the organ to be predicted obtained by the conditional Gaussian model (k×1), is the covariance matrix (k×k) of the shape parameters of the organ to be predicted obtained by the conditional Gaussian model; and is the shape parameter of each training sample (k×1); and yes and The cross-covariance matrix between (k×k); and They are and The covariance matrix (k×k) in the training set; is known hour, The conditional Gaussian model is , where N refers to the Gaussian distribution (the content in the above brackets indicates the dimension of the matrix).

[0042] The conditional Gaussian model is a model established based on the conditional probability relationship between the position, length, width and height of adjacent organs. Its function is that when the center point position, length, width and height of the bounding box of an organ A are known, it can be used to predict the mean and standard deviation of the center point position, length, width and height of the bounding box of its adjacent organ B, so as to know the approximate range of organ B.

[0043] The conditional Gaussian model predicts the shape parameters of the organ to be predicted from the shape parameters of the known organ, and then substitutes the shape parameters of the organ to be predicted obtained by CGM into the shape model. The data set in the model space is obtained. Since the data set is obtained in the normalized model space, it needs to be denormalized to the real physical space of the image to obtain the final prediction result.

[0044] It is necessary to build multiple CGMs, each of which models the position and size correlation between the trunk region and an organ. Using the trunk as a known region, CGMs are used to roughly predict the position information of multiple organs in the trunk region, which is used as a constraint for one-to-one target queries in the subsequent Transformer network.

[0045] S3. Organ localization method based on Transformer network with one-to-one target query

[0046] The Transformer network is a powerful network architecture originally used for natural language processing. Due to its built-in self-attention mechanism, the Transformer is suitable for learning the correlation between objects to improve the performance in image recognition, object detection, segmentation and other fields. The present invention uses the self-attention mechanism of the Transformer network to model the correlation between the position and size of different organs.

[0047] The Transformer model is constructed by improving the state-of-the-art 2D detection Transformer (DETR) network. DETR is the first fully end-to-end 2D image object detector that was recently proposed. It uses Transformers to directly convert image features extracted by CNN into localization results. However, since DETR is used for object detection in natural images, it must handle objects that may not be detected in the current image. Although there are a large number of object categories in natural image datasets, the categories and numbers of objects present in each image are mostly different. Given the N objects defined in the DETR network architecture, the actual number of objects presented in the image is usually much smaller than N. Therefore, DETR must use a bipartite graph matching strategy and then identify the class labels of the detected objects in the image through an object classification network.

[0048] Unlike it, the present invention uses a one-to-one target query architecture to enforce a unique target query for each target organ, so there is no need for a bipartite graph matching module and no target classification branch. In addition, since the positions and sizes of different organs are closely related to each other, the present invention uses a learnable spatial encoding of pixel coordinates to characterize the target position. In this way, the self-attention mechanism of the Transformer network is used to model the geometric correlation between organs, ensuring robust detection of all target structures from the 2D fused projection image. The specific steps are as follows:

[0049] S31. Feature extraction of 2D fused projection images

[0050] Each fused image in the coronal and sagittal directions is sent to the CNN network for image feature extraction, and after dimension reduction mapping into a sequence, the prediction results of CGM are added and input into the Transformer.

[0051] S32. Constructing a Transformer model for one-to-one target query

[0052] The feature sequence extracted by CNN and its corresponding spatial position encoding are sent to the encoder module in Transformer for global correlation calculation. For the decoder module, N target queries are used for N organs to be detected. Both the N target queries and the encoder output are used as inputs to the decoder. Under the combined effect of the encoder and CGM prediction, each projected image obtains N sets of 2D bounding boxes.

[0053] S4. 3D back-projection of 2D projection image detection results

[0054] The improved Transformer model is used to perform steps S2 and S3 on the coronal and sagittal projection fusion images respectively. The output of the Transformer model is the 2D bounding box of all target organs in each projection view, which is specifically expressed as follows:

[0055] Let x, y, and z be the pixel coordinates in three spatial dimensions, and the bounding box of the coronal projection is expressed as The bounding box of the sagittal projection is represented as

[0056] Finally, the coronal and sagittal bounding boxes are back-projected to obtain the three-dimensional bounding box of the organ

[0057] In summary, the present invention is suitable for automatic positioning and detection of multiple organs in multimodal medical images. It mainly utilizes the conditional Gaussian model and Transformer to jointly model the geometric correlation between organs and the information complementarity of multimodal images to efficiently and accurately perform automatic organ positioning.

[0058] Beneficial effects of the present invention: The present invention utilizes the conditional Gaussian model and the self-attention mechanism of the Transformer network to simulate the correlation between the position and size of organs. The one-to-one target query architecture enforces a unique target query for each target organ, without the need for bipartite graph matching; the order of the query is the predicted category, so no classification is required, which simplifies the network structure, reduces redundant calculations, and has a faster learning convergence speed. Before performing organ detection, the 3D multimodal image is projected onto two orthogonal 2D planes, and then the complementary information from the multimodal image is combined through a multimodal fusion method. Finally, the obtained 2D bounding box is back-projected to obtain a 3D bounding box, reducing the computational burden and obtaining a more robust organ localization result. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 It is a flow chart of the positioning method described in the present invention.

[0060] Figure 2 The figure is a schematic diagram of the three-dimensional bounding box of multiple organs using a dual-modality PET / CT image as an example; (a) is the coronal plane, (b) is the sagittal plane, and (c) is the transverse plane.

[0061] Figure 3 This diagram uses dual-modality PET / CT images as an example to illustrate the two-dimensional projection and image fusion process.

[0062] Figure 4 The flowchart of the multi-organ localization method of the present invention is explained by taking dual-modality PET / CT images as an example. DETAILED DESCRIPTION

[0063] The multi-modal medical image multi-organ localization method based on one-to-one target query Transformer proposed in this invention is as follows Figure 1 As shown, the purpose of the present invention is to determine Figure 2 The three-dimensional bounding boxes of multiple organs of the human body are shown. The present invention is further described below in conjunction with specific implementation methods.

[0064] S1. 2D projection and image fusion of 3D multimodal images

[0065] S11. Data preprocessing

[0066] First, the finest voxel size among multiple modal images is selected to resample all modalities; then the resampled images are center-cropped to keep the spatial resolution and image size consistent in the dataset.

[0067] S12. Two-dimensional projection and grayscale normalization of multimodal images

[0068] In order to reduce the amount of calculation and the difficulty of positioning, each modality is projected using a projection method that maintains the image characteristics, thereby obtaining a two-dimensional coronal projection and a sagittal projection. Specifically,

[0069] For functional modality images (such as PET images) in multimodal images, maximum density projection is used to continue to highlight organs and tissues with high metabolism. However, the projection image directly obtained by using MIP will be too dark due to the highlighted organs (such as the bladder). For subsequent accurate detection, it is necessary to normalize the grayscale. Specifically, a normalization window of [g(p1), g(p2)] is used, where g(p1) and g(p2) represent grayscale thresholds, which exclude pixels in the image that are lower than the p1 percentage and higher than the p2 percentage. Experimental findings and Providing robust performance.

[0070] For anatomical modality images (such as CT images) in multimodal images, a grayscale normalization window with a Hounsfield Unit (HU) of [-1000, 1000] is used. Experiments show that the algorithm is insensitive to the grayscale window of CT.

[0071] S13. Image fusion of multimodal projection

[0072] The projection images of each modality are normalized to the grayscale range of [0,255], and then multimodal information fusion is performed according to the fusion image calculation formula:

[0073]

[0074] Among them, α u represents the weight factor, and I u Represents the projection view of the u-th modality image.

[0075] Taking dual-modality PET / CT images as an example, experimental results show that the weight factor of PET produces consistent performance in the value range of [0.3, 0.7]. Each projection view is fused together with a weight of α = 0.5 to generate two fused images of the coronal and sagittal views, respectively, as shown in Figure 3 shown.

[0076] S2. Building a rough prediction model of position correlation between organs through CGM

[0077] A key step in organ localization in multimodal images is to statistically model the correlations between positions and shapes of organs.

[0078] First, we need to obtain a dataset of 3D bounding boxes of organs. The 3D bounding boxes of organs are manually annotated by doctors in multimodal images. We need to obtain the upper and lower limits of the x, y, and z coordinates of the 3D bounding boxes, and then convert them into the center point coordinates and length, width, and height (x, y, z, l x , l y , l z ). Then, the data representing the position and size of the organ in the image space are normalized to the model space through the generalized Protozoa transformation. Then, principal component analysis is performed in the model space, and the principal components whose main features account for more than 90% of all features in the data set are selected to complete the statistical modeling of the organ bounding box based on PCA.

[0079] The steps of the present invention are specifically described by taking the known trunk area as an example to predict an unknown organ. Each training sample in the training set has 6 feature values, namely the center point coordinates and length, width and height of the three-dimensional bounding box, forming a training set (m×n, m is the number of training samples, n=6).

[0080] When predicting organs in the trunk region, the training set of the trunk region is first normalized to obtain the normalized training set X(m×n), and PCA is performed to obtain the eigenvector matrix Φ s (n×k), and then according to Substitute the eigenvector matrix Φ obtained by PCA on the normalized training set s And the normalized training set X to obtain the shape parameter b of each training sample s (k×1). Then the training set of organ bounding boxes is normalized (in the training set, the feature value of each training sample is subtracted from the mean of the corresponding trunk area, and then divided by the standard deviation of the trunk area), and the shape parameter b of the organ bounding box of each training sample is obtained according to the previous steps s (k×1).

[0081] Substitute the obtained shape parameters of the trunk area and organ frame into the CGM formula, the formula is as follows:

[0082]

[0083]

[0084]

[0085] Thus, the shape parameter b of the organ bounding box predicted by the torso region is obtained s According to the shape model Substitute the normalized mean and eigenvector matrix of the organ bounding box training set to obtain s, and then denormalize it to the physical space according to the mean and standard deviation of the torso area to obtain the final prediction result.

[0086] The test set and the training set do not overlap, but the test data must be processed with the same standards as the training data. The trunk area of ​​the test set is used to complete the prediction of the 3D bounding box of the unknown multiple organs, and finally a 2D projection is performed to obtain the rough prediction of the 2D bounding box corresponding to the coronal projection and sagittal projection.

[0087] S3. Organ localization method based on Transformer network with one-to-one target query

[0088] The Transformer model in the present invention is constructed by improving the DETR network.

[0089] Unlike natural object detection, organs detected in medical images have strong anatomical constraints. In most cases, all anatomical structures of interest are present in each target image, and there is only one target object for each structure. We use this anatomical constraint to simplify the network architecture. Given N organs to be detected, only N target queries are defined, specifying that each query corresponds to one target organ, so no bipartite graph matching strategy is required. The order of each query naturally corresponds to the category of each organ, so no additional classification is required. The specific detection process is as follows:

[0090] The fused image is first sent to the CNN network for image feature extraction. We selected the ResNet-50 structure. In experiments with different numbers of CNN layers, the best performance was achieved with 50 layers. The image features are mapped to sequences after dimensionality reduction and the corresponding position encoding is added before they can be used as the input of the encoder module in the Transformer part. In addition, the rough prediction results of the CGM organs are also used as a channel input for subsequent target position and size constraints.

[0091] The self-attention layer in the encoder can aggregate information from each element of the input sequence and update the information of each element, so that the network can perform global correlation calculations and be applicable to long sequences, playing an important role.

[0092] In addition, in the input part of the decoder module, we only define N target queries, specifying that each query corresponds to a target organ, and the order of each query corresponds to the category of each organ. Under the combined effect of the decoder and CGM prediction results, N outputs are obtained, corresponding to the positions of N organs.

[0093] The geometric correlations between organs are jointly modeled through the attention mechanism of the Transformer network and CGM, ensuring robust detection of all target structures from the 2D fused projection images.

[0094] S4. 3D back-projection of 2D projection image detection results

[0095] The same detection process is performed on the coronal and sagittal projection fusion images. The output of the Transformer model is the 2D bounding box of all target organs in the coronal and sagittal projection fusion images. Among them, the coronal projection can obtain the length and height of the 3D bounding box, and the sagittal projection can obtain the width and height of the 3D bounding box. Then the maximum range of the height of the two projection planes is taken, and the detection results in the two directions are back-projected to obtain the final 3D bounding box of the organ.

[0096] Figure 4 This is a flowchart of the multi-organ localization method based on the Transformer network of one-to-one target query, which is illustrated using dual-modality PET / CT images as an example.

Claims

1. A multi-modal medical image multi-organ localization method based on one-to-one target query Transformer, characterized in that: The method comprises the following steps: S1. 2D projection and image fusion of 3D multimodal images S11. Data preprocessing Perform preprocessing of the dataset to normalize the spatial resolution and image size; S12. Two-dimensional projection and grayscale normalization of multimodal images S121, projecting each modality image using a projection method that maintains image characteristics to obtain a two-dimensional coronal projection and a sagittal projection; S122, for each modality projection image, an appropriate threshold range is selected to normalize the grayscale of the image to the grayscale range of [0, 255]; S13. Image fusion of multimodal projection Each projection view is fused together to generate two fused images of the coronal and sagittal views respectively; the fused image calculation formula is: Among them, α u represents the weight factor, and I u represents the projection view of the u-th modality image; S2. Building a rough prediction model of position correlation between organs through CGM S21. Constructing the shape vector of the organ The center point coordinates and length, width and height of the organ's three-dimensional bounding box form a training set; each training sample in the training set constructs a shape vector, let s i is the shape vector of training sample i, i.e. s i =(x, y, z, l x , l y , l z ); S22, Generalized Plattsburgh Transform The shape vector of each training sample in the image space is normalized to the model space through the generalized Protozoa transform, and then subsequent operations are performed in the model space; S23. Principal component analysis All s after generalized Protozoa transformation i As the training set, the statistical shape model method is used to model the position and size changes of the organs, and the shape model is expressed as follows: in, is the average shape of the training set, which is the average value of the features of the selected training set; Φ s is the eigenvector matrix obtained by principal component analysis of the training set, the subscript s represents the shape, and the k eigenvectors represent the k organ position and size change patterns learned from the training set; b s is the shape parameter, b s The k elements in Φ represent s The weight of the k deformation modes in the training set superimposed with the average shape; s represents the final shape model, b s Affect s by adjusting b s The value of can control the deformation of the model; According to formula (2), the shape parameter of each training sample is inversely calculated as follows: in, is the transpose of the eigenvector matrix obtained after principal component analysis of the training set; S24. Constructing a conditional Gaussian model set up and Represent the shape parameters of two adjacent organs A and B respectively. When it is known hour, The conditional Gaussian model CGM description is shown in formulas (4)-(6): in, is the mean of the shape parameters of the organ to be predicted obtained by the conditional Gaussian model, with a dimension of k×1; is the covariance matrix of the shape parameters of the organ to be predicted obtained by the conditional Gaussian model, with a dimension of k×k; and is the shape parameter of each training sample, with dimension k×1; and yes and The cross-covariance matrix between them has dimension k×k; and They are and The covariance matrix in the training set has dimension k×k; is known hour, The conditional Gaussian model, N refers to the Gaussian distribution; Substitute the shape parameters of the organ to be predicted obtained by CGM into the shape model In the above example, we obtain the dataset of the model space, and then denormalize it to the real physical space of the image to obtain the final prediction result. S3. Organ localization method based on Transformer network with one-to-one target query S31. Feature extraction of 2D fused projection images Each fused image in the coronal and sagittal directions is sent to the CNN network for image feature extraction, and the dimensionality is reduced and mapped into a sequence, and then the prediction results of CGM are added and input into the Transformer together; S32. Constructing a Transformer model for one-to-one target query The feature sequence extracted by CNN and its corresponding spatial position encoding are sent to the encoder module in Transformer for global correlation calculation; for the decoder module, N target queries are used for N organs to be detected respectively; N target queries and encoder outputs are used as inputs to the decoder. Under the combined effect of the encoder and CGM prediction, each projected image obtains N sets of 2D bounding boxes; S4. 3D back-projection of 2D projection image detection results Steps S2 and S3 are performed on the coronal and sagittal projection fusion images respectively. The output of the Transformer model is the 2D bounding box of all target organs in each projection view, which is specifically expressed as follows: Let x, y, and z be the pixel coordinates in three spatial dimensions, and the bounding box of the coronal projection is expressed as The bounding box of the sagittal projection is represented as Finally, the coronal and sagittal bounding boxes are back-projected to obtain the organ's 3D bounding box.

2. According to claim 1, a multi-modal medical image multi-organ localization method based on one-to-one target query Transformer is characterized in that: The process of data preprocessing in step S11 is as follows: First, the multimodal images are resampled to a uniform voxel size based on the finest spatial resolution among the multiple modal images. Then, the image size is selected and the resampled images are center cropped to remove surrounding background pixels while retaining the human body area.

3. The method for multi-organ localization in multi-modal medical images based on a one-to-one target query Transformer according to claim 1 or 2, characterized in that: In step S121, the projection method of the modal image is specifically as follows: for functional modal images, maximum density projection is selected to ensure good contrast of highly metabolic organs; for anatomical modal images, average density projection is selected to maintain tissue contrast.

4. The method for multi-organ localization in multi-modal medical images based on a one-to-one target query Transformer according to claim 1 or 2, characterized in that: In step S122, the threshold range specifically uses a normalized window of [g(p1), g(p2)], where g(p1) and g(p2) represent grayscale thresholds that exclude pixels in the image that are lower than the p1 percentage and higher than the p2 percentage.

5. According to claim 3, a multi-modal medical image multi-organ localization method based on one-to-one target query Transformer is characterized in that: In step S122, the threshold range specifically uses a normalized window of [g(p1), g(p2)], where g(p1) and g(p2) represent grayscale thresholds that exclude pixels in the image that are lower than the p1 percentage and higher than the p2 percentage.