Multi-modal mr scan optimization method and system based on self-supervised knowledge transfer

Abstract

The application discloses a kind of multi-modal MR scanning optimization method and system based on self-supervised knowledge transfer, wherein the method includes the arrangement combination of several multi-modal 3D image sequences obtained, to obtain MR image scanning sequence combination set;Multi-modal 3D image sequence is extracted and processed to obtain corresponding high-dimensional feature representation;High-dimensional feature representation is fused with MR image scanning sequence combination set, to obtain the global image feature corresponding to each image scanning sequence combination;Each image feature is input into pre-trained image recognition model, to obtain the feature recognition result corresponding to each image scanning sequence combination;Each feature recognition result is comprehensively analyzed, and the image scanning sequence combination corresponding to the analysis result with the highest comprehensive score is used as the optimal scanning sequence combination.The method of the application improves the efficiency of large-scale MR scanning under the premise of ensuring recognition performance.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a multimodal MR scan optimization method and system based on self-supervised knowledge transfer. Background Technology

[0002] In the field of medical image processing, magnetic resonance imaging (MR) has become one of the core technologies for accurately capturing image details of target areas due to its excellent soft tissue resolution, radiation-free characteristics and multimodal imaging capabilities. It is widely used in the image analysis of deep tissues.

[0003] However, existing magnetic resonance imaging (MR) scan processing methods require processing all acquired modal sequences, making feature recognition time-consuming for hours or even longer. Furthermore, redundant sequences (such as auxiliary modalities with anatomical information similar to other current sequences) introduce a large number of invalid features. These invalid features not only dilute key feature information but also lead to deviations in the recognition of key features, preventing the high-resolution advantage of MR technology from being effectively translated into efficient recognition capabilities. In large-scale image recognition scenarios, this severely impacts the efficiency of feature recognition. Summary of the Invention

[0004] This invention provides a multimodal MR scanning optimization method and system based on self-supervised knowledge transfer, which solves the technical problem of how to optimize existing multimodal MR scanning methods to improve the efficiency of large-scale image recognition.

[0005] To address the aforementioned technical problems, this invention provides, in one aspect, a multimodal MR scan optimization method based on self-supervised knowledge transfer, comprising:

[0006] In response to image processing signals, a number of multimodal 3D image sequences are acquired;

[0007] The multimodal 3D image sequences are arranged and combined based on interpretable positional coding technology to obtain an MR image scan sequence combination set, wherein each image scan sequence combination in the MR image scan sequence combination set is different;

[0008] Feature extraction processing is performed on the multimodal 3D image sequences to obtain high-dimensional feature representations corresponding to each multimodal 3D image sequence. The feature extraction is designed to extract spatial and semantic features associated with image recognition.

[0009] The high-dimensional feature representation is fused with the MR image scan sequence combination set to obtain the global image features corresponding to each image scan sequence combination.

[0010] Each of the global image features is input into a pre-trained image recognition model to obtain the feature recognition results corresponding to each combination of image scanning sequences; wherein, the training process of the image recognition model is designed to perform iterative training based on the first loss of the full sequence combination and the real label in the MR image scanning sequence combination set and the second loss of the non-full sequence combination and its neighboring objects;

[0011] The feature recognition results are comprehensively analyzed, and the combination of image scanning sequences corresponding to the analysis result with the highest comprehensive score is taken as the optimal combination of scanning sequences.

[0012] As one preferred embodiment, the arrangement and combination of the multimodal 3D image sequences based on interpretable location coding technology to obtain an MR image scan sequence combination set includes:

[0013] Based on interpretable positional coding technology, the sequence selection logic data of each of the multimodal 3D image sequences are processed by masking rule definition to generate several mask vectors;

[0014] The multimodal 3D image sequences are arranged and combined according to the mask vector to obtain an MR image scan sequence combination set composed of a number of image scan sequence combinations;

[0015] The MR image scan sequence combination set includes a full sequence combination consisting of all multimodal 3D image sequences, and a non-full sequence combination consisting of some multimodal 3D image sequences.

[0016] As one preferred embodiment, the step of performing feature extraction processing on the multimodal 3D image sequences to obtain high-dimensional feature representations corresponding to each of the multimodal 3D image sequences includes:

[0017] Deep feature mining is performed on each of the multimodal 3D image sequences to obtain deep image features corresponding to each of the multimodal 3D image sequences; wherein, the deep feature mining is used to capture features in the multimodal 3D image sequences that are related to key differences in the target image;

[0018] The deep image features are subjected to dimensionality mapping and condensation processing to obtain high-dimensional feature representations corresponding to each of the multimodal 3D image sequences.

[0019] As a preferred embodiment, the process of fusing the high-dimensional feature representation with the MR image scan sequence combination set to obtain the global image features corresponding to each image scan sequence combination includes:

[0020] A mask location dictionary corresponding to the MR image scan sequence combination set is established based on interpretable location coding technology. The mask location dictionary stores a mask vector corresponding to each image scan sequence combination. Each component of the mask vector corresponds to a multimodal 3D image sequence. A component value of 1 indicates that the multimodal 3D image sequence belongs to the current image scan sequence combination, and a value of 0 indicates that it does not belong to it.

[0021] The target mask vector corresponding to any image scan sequence combination in the MR image scan sequence combination set is matched with the mask location dictionary. Based on the component values ​​of the target mask vector, the high-dimensional feature representation corresponding to the component value of 1 is selected from the high-dimensional feature representations corresponding to all multimodal 3D image sequences as the feature to be fused.

[0022] The selected features to be fused are weighted and accumulated to obtain the global image features of the corresponding image scan sequence combination, wherein the weight of each feature to be fused is the value of its corresponding target mask vector component.

[0023] As one preferred embodiment, before inputting each of the global image features into the pre-trained image recognition model, the method further includes:

[0024] Obtain the original multimodal MR image sequence set and its corresponding ground truth labels;

[0025] Feature extraction is performed on each 3D image sequence in the original multimodal MR image sequence set to obtain a historical high-dimensional feature representation;

[0026] Based on the historical high-dimensional feature representation and the real feature label of the image, the first stage of joint training is performed on the pre-constructed initial target image feature prediction model to obtain the first image recognition model;

[0027] The multimodal MR image sequence set is arranged and combined based on interpretable location coding technology to obtain full sequence combined image features and non-full sequence combined image features;

[0028] The first image recognition model is jointly trained in the second stage based on the combined features of the full sequence image and the combined features of the non-full sequence image to obtain a trained image recognition model.

[0029] As one preferred embodiment, the second-stage joint training of the first image recognition model based on the full-sequence combined image features and the non-full-sequence combined image features to obtain a trained image recognition model includes:

[0030] The full sequence combined image features are input into the first image recognition model, the first recognition result corresponding to the full sequence combination is output, and the cross-entropy loss between the first recognition result and the real label is calculated as the teacher loss.

[0031] Determine the learning object in the Hamming distance neighborhood of each non-full sequence combined image feature, wherein the learning object is an MR sequence combination that adds a 3D image sequence feature to the non-full sequence combined image feature and contains all the 3D image sequence features of the non-full sequence combined image feature;

[0032] The non-full sequence combined image features and their corresponding learning objects are respectively input into the first image recognition model to obtain the second recognition result of the non-full sequence combined image features and the third recognition result of the corresponding learning objects, and the mean squared error loss of the second recognition result and the third recognition result is calculated as the student loss;

[0033] The first image recognition model is iteratively optimized based on the teacher loss and the student loss to obtain a trained image recognition model.

[0034] As one preferred embodiment, the step of comprehensively analyzing each feature recognition result and selecting the image scanning sequence combination corresponding to the analysis result with the highest comprehensive score as the optimal scanning sequence combination includes:

[0035] Based on the first recognition result of the full sequence image scan combination, non-full sequence combinations that have no statistically significant difference in recognition performance from the feature recognition results of each non-full sequence image scan combination are selected to form an initial candidate combination set.

[0036] Determine the safety risk level of each initial candidate combination in the initial candidate combination set, wherein the safety risk level is determined based on the number of MR enhancement sequences in each initial candidate combination, the MR enhancement sequences including MR scan sequences using contrast agents;

[0037] The initial candidate combination set is further screened based on the security risk level to obtain the optimal scanning sequence combination.

[0038] Another aspect of the present invention provides a multimodal MR scan optimization system based on self-supervised knowledge transfer, comprising:

[0039] The acquisition module is used to acquire a number of multimodal 3D image sequences in response to image processing signals;

[0040] The combination module is used to arrange and combine the multimodal 3D image sequences based on interpretable positional coding technology to obtain an MR image scan sequence combination set, wherein the combinations of each image scan sequence in the MR image scan sequence combination set are different.

[0041] The extraction module is used to perform feature extraction processing on the multimodal 3D image sequence to obtain a high-dimensional feature representation corresponding to each multimodal 3D image sequence, wherein the feature extraction is designed to extract spatial features and semantic features associated with image recognition;

[0042] The fusion module is used to fuse the high-dimensional feature representation with the MR image scan sequence combination set to obtain the global image features corresponding to each image scan sequence combination.

[0043] The prediction module is used to input each of the global image features into a pre-trained image recognition model to obtain the feature recognition results corresponding to each combination of image scanning sequences; wherein, the training process of the image recognition model is designed to perform iterative training based on the first loss of the full sequence combination and the real label in the MR image scanning sequence combination set and the second loss of the non-full sequence combination and its neighboring objects;

[0044] The analysis module is used to perform comprehensive analysis on the recognition results of each feature, and to select the combination of image scanning sequences corresponding to the analysis result with the highest comprehensive score as the optimal combination of scanning sequences.

[0045] In another aspect, the present invention provides a multimodal MR scan optimization device based on self-supervised knowledge transfer, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements the multimodal MR scan optimization method based on self-supervised knowledge transfer as described above.

[0046] In another aspect, the present invention provides a computer-readable storage medium storing a computer program, wherein when the device containing the computer-readable storage medium executes the computer program, it implements the multimodal MR scanning optimization method based on self-supervised knowledge transfer as described above.

[0047] Compared with the prior art, the beneficial effects of the embodiments of the present invention are at least one of the following:

[0048] 1) This invention uses interpretable positional encoding technology to arrange and combine multimodal 3D image sequences. This allows for the generation of a unique, structurally clear MR image scan sequence set while preserving the spatial correlation between different modalities. This design solves the problems of information redundancy or loss of positional information that often occur in traditional sequence combinations, providing a structured and traceable sequence foundation for subsequent feature fusion and avoiding subsequent recognition errors caused by disordered sequence arrangement. Simultaneously, the feature extraction stage specifically extracts spatial and semantic features relevant to image recognition. Compared to single-dimensional feature extraction, this approach accurately captures the structural details of the image and uncovers the semantic information it contains. This makes high-dimensional feature representation more aligned with the core needs of MR image recognition, significantly enhancing the supporting value of features for subsequent recognition tasks and laying the foundation for the effectiveness of global image features.

[0049] 2) In the feature processing and model training stages, the fusion processing of high-dimensional feature representation and MR image scan sequence combination sets achieves an organic combination of depth image features and sequence structure information. The generated global image features can completely map the scanning information of each sequence combination, avoiding recognition bias caused by the disconnect between features and sequences, and ensuring that the features of each sequence combination can accurately reflect its scanning value. In addition, the pre-trained image recognition model adopts a dual-loss iterative training of the first loss (full sequence combination and real label) + the second loss (non-full sequence combination and neighborhood object): the first loss anchors the overall recognition accuracy by comparing the full sequence with the real label, ensuring that the model's recognition of the complete scan sequence meets the standard; the second loss optimizes the consistency of local features by constraining the non-full sequence with neighborhood objects, solving the recognition bias problem that is prone to occur in non-full sequences under single-loss training. This dual-loss design allows the feature recognition results output by the model to have both global accuracy and local reliability. The optimal scan sequence combination obtained by the final analysis can not only meet the requirements of high recognition performance, but also provide direction for the optimization of the MR scanning process by selecting efficient sequence combinations. Attached Figure Description

[0050] Figure 1 This is a flowchart illustrating a multimodal MR scanning optimization method based on self-supervised knowledge transfer in one embodiment of the present invention.

[0051] Figure 2 This is a schematic diagram of a multimodal MR scanning optimization system based on self-supervised knowledge transfer in one embodiment of the present invention;

[0052] Figure 3 This is a schematic diagram of a multimodal MR scanning optimization device based on self-supervised knowledge transfer in one embodiment of the present invention;

[0053] Figure label:

[0054] The module consists of: 11. Acquisition module; 12. Combination module; 13. Extraction module; 14. Fusion module; 15. Prediction module; and 16. Analysis module. Detailed Implementation

[0055] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The purpose of providing these embodiments is to make the disclosure of the present invention more thorough and comprehensive. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0056] In the description of this application, the terms "first," "second," "third," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined with "first," "second," "third," etc., may explicitly or implicitly include one or more of that feature. In the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0057] In the description of this application, it should be noted that, unless otherwise expressly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to fixed connections, detachable connections, or integral connections; they can refer to mechanical connections or electrical connections; they can refer to direct connections or indirect connections through an intermediate medium; and they can refer to the internal communication between two components. The terms "vertical," "horizontal," "left," "right," "upper," "lower," and similar expressions used herein are for illustrative purposes only and do not indicate or imply that the device or component referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as limiting the invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.

[0058] In the description of this application, it should be noted that, unless otherwise defined, all technical and scientific terms used in this invention have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in this specification is for the purpose of describing specific embodiments only and is not intended to limit the invention. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.

[0059] One embodiment of the present invention provides a multimodal MR scan optimization method based on self-supervised knowledge transfer. For details, please refer to [link to relevant documentation]. Figure 1 , Figure 1 The diagram shown is a flowchart of a multimodal MR scanning optimization method based on self-supervised knowledge transfer in one embodiment of the present invention, which includes steps S1-S6:

[0060] Step S1: In response to the image processing signal, acquire a number of multimodal 3D image sequences;

[0061] The image processing signal source is based on the image recognition task requirements of existing datasets, and the multimodal 3D image sequences are extracted from existing public / institutional labeled datasets. The dimensions of each acquired multimodal 3D image sequence are uniformly standardized to a predefined standard format, represented as follows: ,in, For the real number field, For image height, For image width, For image depth, For example, the number of channels can be... , , The default value is 512×512×32. Based on sequence mode, the single-channel structure sequence is determined. =1, Multichannel Functional Sequence =3, to ensure that the dimensions of each sequence data are consistent.

[0062] In this embodiment, a number of 3DMR sequences of different modalities need to be acquired to form the core set of multimodal 3D image sequences. Each modal sequence needs to cover different feature dimensions required for image recognition (such as structural features, tissue signal difference features, etc.) to avoid information loss caused by a single modality. The multimodal 3D image sequence set can be represented as follows: This means that 8 different modal 3DMR sequences need to be obtained.

[0063] It should be noted that each multimodal 3D image sequence focuses on a specific region, which must cover the key structural range required for image recognition. This ensures that each 3D image sequence can fully capture the subtle spatial features and tissue signal differences within the region, avoiding the inability of subsequent image recognition models to extract effective features due to scanning range deviations.

[0064] Step S2: Based on interpretable positional coding technology, the multimodal 3D image sequences are arranged and combined to obtain an MR image scan sequence combination set, wherein each image scan sequence combination in the MR image scan sequence combination set is different;

[0065] It should be noted that interpretable location coding technology is a coding method used to clarify the location identification and selection logic of multimodal 3D image sequences. Its function is to assign a unique location index to each sequence and transform the logic of whether a sequence is selected into a combination into a traceable coding form, so as to ensure that each subsequent sequence combination can clearly correspond to the specific original sequence.

[0066] Preferably, in one embodiment of the present invention, the step of arranging and combining the multimodal 3D image sequences based on interpretable positional encoding technology to obtain an MR image scan sequence combination set includes:

[0067] Based on interpretable positional coding technology, the sequence selection logic data of each multimodal 3D image sequence is processed by mask rule definition to generate several mask vectors. The sequence selection logic data refers to the binary logic of "whether a certain multimodal 3D image sequence is included in the current combination" (i.e., "selected" or "not selected"), which is the core logic basis for generating sequence combinations. The "mask rule definition processing" is the process of converting the binary logic into standardized numerical codes. In this embodiment, the conversion is achieved through "0-1 mask". The rules are clear: "1" indicates that the sequence at the corresponding position is selected, and "0" indicates that the sequence at the corresponding position is not selected. Each mask must cover all 8 sequence positions to form a mask vector of fixed length.

[0068] Specifically, for the multimodal 3D image sequence obtained in step S1, a unique location index is assigned to establish a "sequence-location" mapping relationship. For example, ... Mapped to position 1 Mapped to position 2... Mapping to position 8 creates 8 fixed sequence position dimensions. This mapping relationship needs to be stored in a temporary database as the basis for subsequent masking rule definitions, ensuring that the positions of each sequence are not repeated or confused.

[0069] Furthermore, based on the established "sequence-position" mapping relationship, a unified masking rule is defined: for the 8 sequence positions, each position is only allowed to take the value "0" or "1", where "1" corresponds to "selecting the multimodal 3D image sequence at this position", and "0" corresponds to "not selecting the multimodal 3D image sequence at this position"; at the same time, it is limited that there must be at least one "1" in the mask vector (i.e. at least one sequence is selected) to avoid generating empty combinations that have no practical meaning.

[0070] The multimodal 3D image sequences are arranged and combined according to the mask vector to obtain an MR image scan sequence combination set consisting of a number of image scan sequence combinations; wherein, the MR image scan sequence combination set includes a full sequence combination consisting of all multimodal 3D image sequences, and a non-full sequence combination consisting of some multimodal 3D image sequences. The mask vector is an 8-dimensional numerical vector generated according to masking rules, in the form of... ,in A value of "0" or "1" corresponds to "not selected" or "selected" in the sequence at position i, respectively; each mask vector uniquely corresponds to one sequence combination, for example, mask vector m=(1,0,0,0,0,0,0,0) corresponds to "selected only". The combination of "" and m=(1,1,0,0,0,0,0,0) corresponds to "selected". and The combination of "".

[0071] Specifically, all mask vectors conforming to the masking rules are generated using a "0-1 combination traversal" method: from a vector containing only one "1" to a vector containing two "1"s (total... =28 kinds), until a vector containing 8 "1"s (1 kind, i.e., the entire sequence combination), finally generating 2 8 -1 = 255 mask vectors, and all vectors together form a mask position dictionary.

[0072] Furthermore, for each mask vector in the mask location dictionary, a vector-to-combination transformation is performed. Specifically, a certain mask vector is read... Judge each position one by one of Value; if =1, then from the image sequence set Selecting multimodal 3D image sequences at corresponding locations ;like =0, then exclude ; All selected These are then combined to form a single image scan sequence. For example, the mask vector 11111111 corresponds to the selected... to The combination of all sequences, i.e., the complete sequence combination; the mask vector 10100000 corresponds to selecting only... and This involves combining sequences, specifically non-complete sequence combinations. The process ensures that each mask vector corresponds to only one combination, and that no sequences within a combination are repeated, thus providing subsequent image recognition models with feature input sources of varying numbers and compositions of sequences.

[0073] The above process yields the MR image scanning sequence combination set in this embodiment of the invention, which is a set composed of all the above image scanning sequence combinations. The set contains 255 different combinations, and each combination has a unique corresponding mask vector (which can be traced through the mask position dictionary M). The set is clearly divided into two types of combinations: full sequence combinations (only 1 type, containing 8 sequences) and non-full sequence combinations (254 types, containing 1-7 sequences).

[0074] Step S3: Perform feature extraction processing on the multimodal 3D image sequence to obtain high-dimensional feature representations corresponding to each multimodal 3D image sequence, wherein the feature extraction is designed to extract spatial features and semantic features associated with image recognition;

[0075] Preferably, in one embodiment of the present invention, the step of performing feature extraction processing on the multimodal 3D image sequences to obtain high-dimensional feature representations corresponding to each of the multimodal 3D image sequences includes:

[0076] Deep feature mining is performed on each of the multimodal 3D image sequences to obtain the deep image features corresponding to each multimodal 3D image sequence. Specifically, deep feature mining refers to the process of extracting multi-layer features from the multimodal 3D image sequences using a deep learning model. In this embodiment, the deep learning model refers to a 3D convolutional neural network. Through deep feature mining, spatial features (such as the three-dimensional shape, location distribution, and structural relationships of targets within the scanned area) and semantic features (such as signal differences between different tissues and distinguishing features between targets and background) related to image recognition can be captured.

[0077] In this embodiment, a 3DCNN image feature encoder needs to be constructed first. Specifically, a convolutional neural network adapted to 3D images is built as the core of feature extraction. Its network structure includes multiple layers of 3D convolutional layers, 3D pooling layers, and activation functions. The 3D convolutional layers capture the spatial correlation of multimodal 3D image sequences (such as the morphological features of the target in the three-dimensional directions of height, width, and depth) through three-dimensional convolutional kernels (such as 3×3×3). The 3D pooling layers (such as max pooling) are used to reduce the feature dimensionality and retain key information. The activation functions enhance the network's ability to fit nonlinear features, ensuring that the deep semantic features required for image recognition can be extracted.

[0078] The multimodal 3D image sequence obtained in step S1 Perform normalization processing, and then normalize the result. Each image is input into a 3DCNN image feature encoder, and the deep image features of the corresponding sequence are calculated and output through forward propagation of the network.

[0079] Furthermore, the deep image features are subjected to dimensionality mapping and condensation processing to obtain high-dimensional feature representations corresponding to each of the multimodal 3D image sequences. Dimensionality mapping and condensation processing refers to the process of transforming high-dimensional, multi-channel deep image features into low-dimensional, compact vector forms through linear projection operations. The core purpose is to reduce the number of parameters and computational complexity in subsequent feature fusion and model training, while retaining key information in the deep image features; the high-dimensional feature representation is the output of this processing.

[0080] Specifically, a linear transformation layer containing the projection matrix and bias terms is constructed, which can be represented as:

[0081]

[0082] in, Let be the projection matrix, with dimension . This is used to map the flattened deep feature vectors to the target dimension. The bias term has the following dimensions: This is used to adjust the feature shift after linear transformation, ensuring that the feature distribution better fits the subsequent model. Let n be the target dimension, and n be the total number of elements after flattening the deep image features. .

[0083] The flattened one-dimensional vector is input into a linear projection layer to perform a linear transformation on the one-dimensional vector. For example, if... =256, then for The matrix, through matrix multiplication vector mapping to vectors, then superimposed ( The offset is adjusted, and the final output dimension is... The generated high-dimensional feature representation is then subjected to a feature variance test (e.g., calculating the variance of the high-dimensional feature representation to ensure it is greater than a preset threshold, avoiding information loss due to overly flat features). If the variance does not meet the threshold, the linear projection layer is adjusted. Value (e.g., increase) (Up to 512) or retrain the 3DCNN encoder to ensure that the high-dimensional feature representation can effectively carry the key features required for image recognition.

[0084] Step S4: The high-dimensional feature representation is fused with the MR image scan sequence combination set to obtain the global image features corresponding to each image scan sequence combination;

[0085] Preferably, in one embodiment of the present invention, the step of fusing the high-dimensional feature representation with the MR image scan sequence combination set to obtain the global image features corresponding to each image scan sequence combination includes:

[0086] A mask location dictionary corresponding to the MR image scan sequence combination set is established based on interpretable location coding technology. This dictionary stores a mask vector corresponding to each image scan sequence combination. Each component of the mask vector corresponds to a multimodal 3D image sequence. A value of 1 indicates that the multimodal 3D image sequence belongs to the current image scan sequence combination, while a value of 0 indicates that it does not. Specifically, for the MR image scan sequence combination set (containing 255 combinations) generated in step S2, a unique mask vector m is assigned to each combination. A key-value pair mapping relationship is established using the combination ID as the key and the mask vector m as the value, forming the mask location dictionary M. To improve interpretability, an additional reverse mapping between the mask vector and the sequence list is stored to ensure rapid tracing of the combination composition and provide a clear logical basis for feature fusion related to image recognition. All mask vectors in the dictionary are traversed to verify whether there are no empty combinations and whether each vector uniquely corresponds to a combination. Duplicate or invalid vectors are removed to ensure the integrity and uniqueness of the dictionary.

[0087] Further, the target mask vector corresponding to any image scan sequence combination in the MR image scan sequence combination set is matched with the mask location dictionary. Based on the component values ​​of the target mask vector, the high-dimensional feature representation corresponding to the component value of 1 is selected from the high-dimensional feature representations corresponding to all multimodal 3D image sequences as the feature to be fused.

[0088] Here, the target mask vector refers to a specific mask vector matched from the mask position dictionary that corresponds to a combination of image scanning sequences, and its dimension is consistent with the vectors in the mask position dictionary. The features to be fused are the set of features selected from the high-dimensional feature representation obtained in step S3 that correspond to the component "1" of the specific mask vector. These features form the basis of the current combined global image features and carry the key information required for image recognition.

[0089] Specifically, for a given combination in the MR image scan sequence combination set, its combination ID is queried in the mask location dictionary M to obtain the corresponding target mask vector. The high-dimensional feature representations of all multimodal 3D image sequences generated in step S3 are then used to establish a mapping relationship between "sequence location and high-dimensional feature". The eight components of the target mask vector are traversed; if a component is 1, the high-dimensional feature at the corresponding location is selected and added to the feature set to be fused; if a component is 0, the high-dimensional feature at that location is excluded.

[0090] Furthermore, the selected features to be fused are subjected to weighted accumulation processing to obtain global image features of the corresponding image scan sequence combination, wherein the weight of each feature to be fused is the value of its corresponding target mask vector component.

[0091] It should be noted that global image features are the final fusion features of a combination of image scanning sequences, which can comprehensively reflect the key features of all sequences within the combination. Weighted accumulation processing refers to the process of summing the selected features to be fused using the components of the target mask vector as weights. The weight value is only 1, which is essentially a direct accumulation of effective features.

[0092] Specifically, the components of the target mask vector are directly used as weights for the corresponding features to be fused. Since the components are only 0 or 1, a weight of 1 indicates that the corresponding feature needs to be fused, while a weight of 0 indicates that it has been filtered out. Vector summation is then performed based on the following formula:

[0093]

[0094] in, For sequence combination The A mask for a sequence, To correspond to the features to be fused.

[0095] Verify whether the generated global image feature dimension is 0. If the dimensions do not match, the filtering process will be checked back.

[0096] Step S5: Input each of the global image features into the pre-trained image recognition model to obtain the feature recognition results corresponding to each combination of image scanning sequences; wherein, the training process of the image recognition model is designed to iteratively train based on the first loss of the full sequence combination and the real label in the MR image scanning sequence combination set and the second loss of the non-full sequence combination and its neighboring objects;

[0097] Preferably, in one embodiment of the present invention, before inputting each of the global image features into the pre-trained image recognition model, the method further includes:

[0098] Obtain the original multimodal MR image sequence set and its corresponding ground truth labels; where the original multimodal MR image sequence set refers to the stored multimodal 3D MR sequence set that meets the requirements of image recognition, containing a large amount of image data of scanned objects, which is used as data support for model training; ground truth labels refer to the ground truth labels required for image recognition corresponding to the historical sequence set, such as whether there are binary labels with specific features in the target region, which are the supervision basis for model training and ensure the accuracy of image recognition results.

[0099] Specifically, historical data that meets the requirements of image recognition is selected from the image database, and the selected historical sequence set is subjected to the same data preprocessing process as step S1, ultimately forming a one-to-one correspondence dataset of "original multimodal MR image sequence set - real label".

[0100] Furthermore, feature extraction is performed on each 3D image sequence in the original multimodal MR image sequence set to obtain a historical high-dimensional feature representation; wherein, the historical high-dimensional feature representation is the result of deep feature mining and dimension mapping condensation performed on each 3D image sequence in the historical sequence set.

[0101] Based on the historical high-dimensional feature representation and the real feature label of the image, the first stage of joint training is performed on the pre-constructed initial target image feature prediction model to obtain the first image recognition model; wherein, the first stage of joint training refers to the initial training of the initial model with the historical high-dimensional feature representation and the real label, so that the model can master the basic image recognition logic; the first image recognition model is the output of this stage, providing the basic network structure for subsequent knowledge distillation training.

[0102] Specifically, for each of the eight sequences in the historical sequence set, the feature extraction process of step S3 is executed one by one. The deep image features are mined by the 3DCNN encoder and then compressed into historical high-dimensional features by the linear projection layer. Finally, each sample forms eight high-dimensional features.

[0103] Furthermore, an initial target image feature prediction model is constructed. The initial model includes a feature fusion layer and an MLP prediction layer. The feature fusion layer can combine and weight the eight historical high-dimensional features according to the entire sequence to form global image features. The MLP prediction layer is constructed according to the following formula, and the output layer uses the sigmoid activation function to output the predicted probability of image recognition.

[0104]

[0105] Among them, For the sigmoid function, For network parameters, The predicted probability is based on a combination of multimodal sequences, and its value range is [range missing]. [0, 1].

[0106] Specifically, the global image features of the entire sequence combination are used as input, the real labels are used as supervision signals, the cross-entropy loss function is used to calculate the loss, and the parameters of the 3DCNN, linear projection layer and MLP of the initial model are iteratively updated by gradient descent until the loss converges (e.g., the loss value is less than 0.01), thus obtaining the first image recognition model.

[0107] The multimodal MR image sequence set is permuted and combined using interpretable location coding technology to obtain full-sequence combined image features and non-full-sequence combined image features. Based on the interpretable location coding technology in step S2, the high-dimensional features of the original multimodal MR image sequence set are permuted and combined to generate features. "Full-sequence combined image features" are global image features fused from 8 high-dimensional features using a mask vector of 11111111; "non-full-sequence combined image features" are global image features fused using other mask vectors (such as 10100000). These two types of features together constitute the training data for knowledge distillation.

[0108] Furthermore, the first image recognition model is jointly trained in the second stage based on the full sequence combined image features and the non-full sequence combined image features to obtain a trained image recognition model.

[0109] Specifically, the second phase of joint training uses a knowledge distillation training strategy that uses full sequence combinations as teachers and non-full sequence combinations as students, so that the recognition performance of non-full sequence combinations is close to that of full sequences.

[0110] Preferably, in one embodiment of the present invention, the step of performing a second-stage joint training on the first image recognition model based on the full-sequence combined image features and the non-full-sequence combined image features to obtain a trained image recognition model includes:

[0111] The full sequence combined image features are input into the first image recognition model, and the first recognition result corresponding to the full sequence combination is output. The cross-entropy loss between the first recognition result and the real label is calculated as the teacher loss. The teacher loss is the cross-entropy loss between the recognition result of the full sequence combined image features and the real label. Since the full sequence information is the most complete, its loss is used to ensure the baseline recognition performance of the model. The student loss is the mean squared error loss between the recognition result of the non-full sequence combined image features and the recognition result of the learning object. It is used to transfer the recognition knowledge of the full sequence to the non-full sequence and narrow the performance gap between the two.

[0112] Specifically, the combined image features of the entire sequence are input into the first image recognition model, and the first recognition result (the predicted probability of image recognition, taking values ​​[0,1]) is output through the MLP prediction layer; the cross-entropy loss is calculated using the first recognition result and the true label as input, and is expressed as:

[0113]

[0114] Where N is the total number of samples, and the loss measures the deviation between the full sequence combination recognition result and the true label.

[0115] Further, a learning object is determined within the Hamming distance neighborhood of each non-full sequence combined image feature, wherein the learning object is an MR sequence combination that adds a 3D image sequence feature to the non-full sequence combined image feature and includes all 3D image sequence features of the non-full sequence combined image feature.

[0116] Here, the Hamming distance neighborhood is the set of reference combinations for non-complete sequence combinations. For a certain non-complete sequence combination (Mask vector), whose neighborhood satisfies That is, combinations within the neighborhood. Compare One more sequence, and yes A subset; the learning object is the neighborhood. The combination within the sequence was used as the identification result for non-complete sequence combination. The reference standard ensures that the recognition logic for incomplete sequences aligns with optimal sequence combinations. For mask dictionary.

[0117] Specifically, for each incomplete sequence combination m, a learning object is determined by traversing the mask position dictionary. Find the Hamming distance neighborhood of m. Combinations within the neighborhood That is The learning objects.

[0118] The non-complete sequence combined image features and their corresponding learning objects are respectively input into the first image recognition model to obtain the second recognition result of the non-complete sequence combined image features and the third recognition result of the corresponding learning objects. The mean squared error loss of the second recognition result and the third recognition result is calculated as the student loss. Specifically, the features of the non-complete sequence combined image features and their learning objects are respectively input into the first image recognition model, and the second recognition result is output ( The predicted probability) and the third identification result ( (Predicted probability). The mean squared errors of the second and third identification results for all non-full sequence combinations are summed, and then averaged:

[0119]

[0120]

[0121]

[0122] in, yes arrive The adaptive weights are obtained by calculating the correlation coefficient using the softmax function to ensure that the weights of the learning objects are consistent with their sum. Relational matching.

[0123] Furthermore, the first image recognition model is iteratively optimized based on the teacher loss and the student loss to obtain a trained image recognition model. Specifically, a total loss function is constructed, expressed as:

[0124]

[0125] in, The distillation intensity coefficient is determined through grid search, such as ( =0.5); Gradient descent is used to iteratively update all parameters of the first image recognition model (3DCNN, linear projection layer, MLP), and the calculation is performed after each iteration. until When the model converges to a preset minimum value range (e.g., the loss fluctuation is less than 0.001 for 10 consecutive iterations), training stops. The model at this point is the pre-trained image recognition model, which can accurately match the real label through the combination of the whole sequence, and can also make the recognition performance of the non-whole sequence combination close to that of the whole sequence.

[0126] After model training is complete, the global image features of the combined image scan sequences generated in step S4 can be processed, and the corresponding feature recognition results can be output. Specifically, the global image features of all the combined image scan sequences generated in step S4 are called, and each global image feature is input into the pre-trained image recognition model one by one. The model performs forward propagation calculations through the MLP prediction layer.

[0127] The feature recognition result corresponding to each global image feature is associated with and stored with the sequence combination ID and mask vector to form a mapping result of sequence combination-global feature-recognition result.

[0128] Step S6: Perform a comprehensive analysis on each of the feature recognition results, and take the combination of image scanning sequences corresponding to the analysis result with the highest comprehensive score as the optimal combination of scanning sequences.

[0129] Before performing two statistical tests, it is necessary to first identify the core indicators for measuring image recognition performance and calculate the indicator values ​​based on the feature recognition results of step S5, providing a data foundation for subsequent tests. According to the technical requirements for comparing the image recognition performance of sequence combinations, feature recognition performance indicators are quantitative indicators that measure the performance of a sequence combination in an image recognition task, and are divided into two categories:

[0130] Second feature recognition performance index: Specifically refers to the image recognition performance index of the whole sequence combination (because the whole sequence information is the most complete, it is used as the performance benchmark), including AUC (comprehensively reflects the image recognition discrimination ability), sensitivity (the proportion of samples that correctly identify the target feature), and specificity (the proportion of samples that correctly identify the target feature); First feature recognition performance index: Specifically refers to the image recognition performance index of non-whole sequence combination, and is consistent with the whole sequence index type.

[0131] Preferably, in a preferred embodiment of the present invention, the step of comprehensively analyzing each of the feature recognition results and selecting the image scanning sequence combination corresponding to the analysis result with the highest comprehensive score as the optimal scanning sequence combination includes:

[0132] Based on the first recognition result of the full sequence image scan combination, non-full sequence combinations with no statistically significant difference in recognition performance from the feature recognition results of each non-full sequence image scan combination are selected to form an initial candidate combination set. Among them, the full sequence image scan combination refers to the combination containing all 8 multimodal 3DMR sequences, that is, the combination with the corresponding mask vector "11111111". Its first recognition result is the feature recognition result output by the pre-trained image recognition model for the full sequence combination, which is specifically quantified as recognition performance indicators, including but not limited to sensitivity, specificity and AUC value (area under the receiver operating characteristic curve).

[0133] Furthermore, the recognition performance of each sequence combination was analyzed using the statistical methods Delong test and chi-square test. The feature recognition results of all non-full sequence combinations, i.e., combinations not containing all eight MR sequences, such as mask vectors "11000000" and "00111000", were examined. Specifically, the Delong test was used to verify the consistency of image recognition discrimination. The Delong test determined whether the AUC_m value of the non-full sequence was significantly different from the AUC_full value of the full sequence. If the difference was not statistically significant, it indicated that the image recognition discrimination ability of the non-full sequence was comparable to that of the full sequence. The chi-square test determined whether the sensitivity and specificity of the non-full sequence were significantly different from those of the full sequence. If both were not significantly different, it indicated that the image recognition accuracy of the non-full sequence was comparable to that of the full sequence.

[0134] If a non-full sequence combination passes the above two tests, and all the differences between the recognition performance indicators and the full sequence combination satisfy P>0.05 (i.e. there is no statistical difference, indicating that the diagnostic efficacy of the two is comparable), then the non-full sequence combination is included in the initial candidate combination set.

[0135] The safety risk level of each initial candidate combination in the initial candidate combination set is determined, wherein the safety risk level is determined based on the number of MR enhancement sequences in each initial candidate combination. The MR enhancement sequences include MR sequences requiring intravenous contrast agent injection. Specifically, some MR sequence scans must use contrast agents, and the use of contrast agents carries certain risks, especially unsuitable for patients with renal insufficiency or those at high risk of contrast agent allergy, such as elderly patients and patients with chronic kidney disease. Therefore, a safety risk assessment is also required for the MR enhancement sequences containing contrast agents in the initial candidate combination set. In this embodiment, initial candidate combinations containing 0 MR enhancement sequences (i.e., all sequences are non-enhancing sequences) are determined to be of low risk; initial candidate combinations containing one or more MR enhancement sequences (i.e., requiring contrast agent injection) are determined to be of high risk. The safety risk coefficient is expressed by the formula:

[0136]

[0137] in, This refers to the safety risk coefficient.

[0138] Furthermore, for each combination in the initial candidate set, the sequence types it contains are checked one by one, and the corresponding security risk level is labeled. Based on the security risk level, the initial candidate combination set is further screened to obtain the optimal scanning sequence combination. Specifically, the standardized recognition performance of each combination in the candidate sequence combination set is calculated, expressed as:

[0139]

[0140] in, For sequence combination Standardized recognition performance The second recognition performance of candidate sequence combinations, This represents the first recognition performance for standard full sequence combinations.

[0141] The cost-benefit ratio of each sequence combination is calculated based on the security risk coefficient and standardized identification effectiveness, and is expressed as follows:

[0142]

[0143] in, For sequence combination Cost-effectiveness ratio Weighting coefficients for security ( ), used to adjust security The extent of the impact.

[0144] Finally, select from the candidate sequence combination set The sequence combination with the largest value is taken as the optimal scan sequence combination, and is represented as:

[0145]

[0146] in, For the optimal scan sequence combination, This is a set of candidate sequence combinations.

[0147] The present invention also provides a specific embodiment of a multimodal MR scan optimization method based on self-supervised knowledge transfer to illustrate the beneficial effects of this solution.

[0148] In response to the image processing signal, which originates from the start command of the multimodal MR image feature recognition task, such as the trigger signal generated after initiating the head and neck image key structural feature extraction task in a medical image analysis platform, multimodal 3D image sequences are extracted from existing open-source head and neck MRI datasets, and four core modal sequences from the dataset are selected to form a sequence set. ,in This is a T1-weighted imaging sequence, focusing on detailed visualization of the nasopharyngeal anatomy. This is a T2-weighted imaging sequence, focusing on the signal difference between tumor tissue and normal soft tissue. To enhance T1-weighted imaging sequences, focusing on the tumor's blood supply characteristics and enhancement patterns, contrast agent enhancement allows for a clearer distinction between the tumor parenchyma and surrounding non-enhancing tissue. As diffusion-weighted imaging sequences, focusing on the diffusion and movement characteristics of water molecules in the microstructure of tissues, these sequences were preprocessed using the SimpleITK tool, and all sequences were uniformly resampled to a dimension of 512×512×32 to ensure that the data format of each sequence was consistent.

[0149] Furthermore, based on interpretable positional encoding technology, the sequence set S is permuted and combined. First, a unique positional index is assigned to each modal sequence, where, Corresponding position 1 Corresponding position 2 Corresponding position 3 For position 4, define the masking rules: generate a 4-dimensional mask vector. A value of 1 for each component indicates that the sequence at that position is included in the combination, and a value of 0 indicates that it is not included. The vector must contain at least one 1 to avoid empty combinations. Based on this, generate all non-empty mask vectors, for example, (1,0,0,0) (containing only 1). (1,1,0,0) (inclusive) and There are 15 types of mask vectors, including (1,1,1,1) (containing all 4 types of sequences, i.e., the complete sequence combination). Each mask vector corresponds to an image scan sequence combination, which together constitute the MR image scan sequence combination set.

[0150] Feature extraction was performed on each modality sequence. A 3D convolutional neural network (3DCNN) was constructed as the feature extractor. This network consists of three 3D convolutional layers (3×3×3 kernel size), two 3D max pooling layers (2×2×2 pooling kernel size), and a ReLU activation function. The preprocessed sequence was then processed... to The data are input into the network, and key features of each sequence are captured through deep feature mining (such as...). Structural contour features, The signal difference characteristics are obtained to obtain the deep image features corresponding to each sequence. Then, the deep image features are dimensionally mapped and condensed through a linear projection layer (projection matrix dimension is 64×256), compressing them into a 256-dimensional high-dimensional feature representation to ensure that the high-dimensional feature dimension of each sequence is consistent.

[0151] A mask position dictionary is established based on interpretable positional coding technology. The dictionary stores the mask vectors corresponding to 15 combinations, such as the combination "containing". and "Corresponds to the mask vector (1,0,1,0). For any combination in the combination set (such as "containing..."), and The function matches the target mask vector (1,0,1,0) from the dictionary and filters high-dimensional features based on the values ​​of the vector components (component 1 corresponds to...). and High-dimensional features, components with 0 are excluded. and The high-dimensional features are used to obtain the feature set to be fused, and then the features in this set are weighted and accumulated (the weight is 1 for the corresponding component). High-dimensional features + The high-dimensional features are used to obtain the global image features corresponding to the combination, and the global image features of all 15 combinations are generated in this way.

[0152] Furthermore, multimodal 3D sequences and corresponding feature annotations from the MRI dataset are obtained, where the annotations are feature labels of key structures in the sequences. The same feature extraction process described above is performed on historical sequences to obtain historical high-dimensional features. The historical high-dimensional features and feature annotations are used to train the initial 3DCNN+MLP model (MLP contains 2 fully connected layers) in the first stage. The global image features combined from the entire sequence are used as input, and the feature annotations are used as supervision signals. The cross-entropy loss function is used to iteratively optimize the model parameters until the loss converges (loss value is less than 0.01), thus obtaining the first image recognition model.

[0153] In the second training stage, the global image features of the full sequence combination generated from the historical dataset are input into the first model, outputting the first recognition result. The cross-entropy loss between the first model and the feature annotation is calculated as the teacher loss. For each non-full sequence combination (e.g., (1,0,0,0)), the learning objects in its Hamming distance neighborhood are determined (i.e., combinations that have one more sequence than the combination and contain all of its sequences, such as (1,1,0,0), (1,0,1,0), (1,0,0,1)). The global image features of the non-full sequence combination and its learning objects are input into the first model, obtaining the second and third recognition results respectively. The mean squared error loss of both is calculated as the student loss. A total loss function is constructed (total loss = teacher loss + 0.5 × student loss, where 0.5 is the distillation intensity coefficient). The model is iteratively optimized until the total loss converges, resulting in a pre-trained image recognition model.

[0154] The 15 previously generated combinations of global image features are input into the pre-trained model one by one, and the feature recognition results corresponding to each combination are output (represented by feature matching probability, with a value range of [0,1]).

[0155] Finally, the feature recognition results are analyzed to determine the optimal combination. Specifically, the recognition results of the full sequence combination (e.g., feature matching probability 0.92, AUC value 0.95) are used as a benchmark. The Delong test is used to screen combinations among the non-full sequence combinations whose recognition performance (feature matching probability, AUC value) is not statistically different from that of the full sequence (P>0.05). For example, (1,0,1,0), (0,1,1,0), and (1,1,1,0) are selected to form an initial candidate set. Then, the safety risk level of each candidate combination is determined (based on the number of MR-enhanced sequences). For the enhancement sequence, combinations containing 0 enhancement sequences are considered low-risk, and combinations containing 1 enhancement sequence are considered high-risk. Among them, (1,0,1,0) is low-risk, and (0,1,1,0) and (1,1,1,0) are high-risk.

[0156] Calculate the cost-benefit ratio of each candidate combination (cost-benefit ratio = safety factor × standardized recognition performance, where the safety factor is 1.0 for low risk and 0.6 for high risk, and the standardized recognition performance is the ratio of the recognition performance of the candidate combination to the recognition performance of the entire sequence). For example, the cost-benefit ratio of (1,0,1,0) is 1.0 × (0.91 / 0.92) = 0.99, the cost-benefit ratio of (0,1,1,0) is 0.6 × (0.90 / 0.92) = 0.59, and the cost-benefit ratio of (1,1,1,0) is 0.6 × (0.92 / 0.92) = 0.60. Select (1,0,1,0) as the optimal scanning sequence combination, which reduces the number of scanning sequences and eliminates the risk of enhanced sequence correlation while ensuring feature recognition performance, thus effectively improving MR scanning efficiency.

[0157] Another embodiment of the present invention provides a multimodal MR scan optimization system based on self-supervised knowledge transfer. For details, please refer to [link to relevant documentation]. Figure 2 , Figure 2 The diagram shown illustrates a multimodal MR scanning optimization system based on self-supervised knowledge transfer, as described in one embodiment of the present invention, comprising:

[0158] Acquisition module 11 is used to acquire a number of multimodal 3D image sequences in response to image processing signals;

[0159] Combination module 12 is used to arrange and combine each of the multimodal 3D image sequences based on interpretable position coding technology to obtain an MR image scan sequence combination set, wherein each image scan sequence combination in the MR image scan sequence combination set is different;

[0160] Extraction module 13 is used to perform feature extraction processing on the multimodal 3D image sequence to obtain high-dimensional feature representations corresponding to each multimodal 3D image sequence, wherein the feature extraction is designed to extract spatial features and semantic features associated with image recognition;

[0161] The fusion module 14 is used to fuse the high-dimensional feature representation with the MR image scan sequence combination set to obtain the global image features corresponding to each image scan sequence combination.

[0162] The prediction module 15 is used to input each of the global image features into a pre-trained image recognition model to obtain the feature recognition results corresponding to each combination of image scanning sequences; wherein, the training process of the image recognition model is designed to perform iterative training based on the first loss of the full sequence combination and the real label in the MR image scanning sequence combination set and the second loss of the non-full sequence combination and its neighboring objects.

[0163] Analysis module 16 is used to perform comprehensive analysis on each of the feature recognition results, and to take the image scanning sequence combination corresponding to the analysis result with the highest comprehensive score as the optimal scanning sequence combination.

[0164] Another embodiment of the present invention provides a multimodal MR scanning optimization device based on self-supervised knowledge transfer. For details, please refer to [link to relevant documentation]. Figure 3 , Figure 3This diagram illustrates a multimodal MR scan optimization device based on self-supervised knowledge transfer in one embodiment of the present invention. It is a structural block diagram of the multimodal MR scan optimization device based on self-supervised knowledge transfer provided in this embodiment. The multimodal MR scan optimization device based on self-supervised knowledge transfer provided in this embodiment includes a processor 21, a memory 22, and a computer program stored in the memory 22 and configured to be executed by the processor 21. When the processor 21 executes the computer program, it implements the steps described in the above embodiment of the multimodal MR scan optimization method based on self-supervised knowledge transfer, for example... Figure 1 The steps S1 to S6 described above; or, when the processor 21 executes the computer program, it implements the functions of each module in the above-described device embodiments, such as the acquisition module 11.

[0165] For example, the computer program may be divided into one or more modules, which are stored in the memory 22 and executed by the processor 21 to complete the present invention. The one or more modules may be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program in the self-supervised knowledge transfer-based multimodal MR scanning optimization device.

[0166] The multimodal MR scanning optimization device based on self-supervised knowledge transfer may include, but is not limited to, a processor 21 and a memory 22. Those skilled in the art will understand that the schematic diagram is merely an example of a multimodal MR scanning optimization device based on self-supervised knowledge transfer and does not constitute a limitation on the device. It may include more or fewer components than illustrated, or combine certain components, or use different components. For example, the multimodal MR scanning optimization device based on self-supervised knowledge transfer may also include input / output devices, network access devices, buses, etc.

[0167] The processor 21 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor. The processor 21 is the control center of the self-supervised knowledge transfer-based multimodal MR scanning optimization device, connecting all parts of the device via various interfaces and lines.

[0168] The memory 22 can be used to store the computer program and / or modules. The processor 21 implements various functions of the multimodal MR scanning optimization device based on self-supervised knowledge transfer by running or executing the computer program and / or modules stored in the memory 22 and calling the data stored in the memory 22. The memory 22 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, phonebook, etc.). In addition, the memory 22 may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0169] The modules integrated into the multimodal MR scanning optimization device based on self-supervised knowledge transfer, if implemented as software functional units and sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc.

[0170] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0171] Accordingly, embodiments of the present invention provide a computer-readable storage medium comprising a stored computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to perform steps in the multimodal MR scan optimization method based on self-supervised knowledge transfer as described in the above embodiments, for example... Figure 1 Steps S1 to S6 as described above.

[0172] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

Claims

1. A method for multi-modality MR scan optimization based on self-supervised knowledge transfer, the method comprising: The method comprises: in response to an image processing signal, obtaining a plurality of multi-modal 3D image sequences; arranging and combining each of the multi-modal 3D image sequences based on an interpretable position coding technology to obtain an MR image scan sequence combination set, wherein each image scan sequence combination in the MR image scan sequence combination set is inconsistent; performing feature extraction processing on the multi-modal 3D image sequences to obtain high-dimensional feature representations corresponding to each of the multi-modal 3D image sequences, wherein the feature extraction is designed to extract spatial features and semantic features associated with image recognition; fusing the high-dimensional feature representations with the MR image scan sequence combination set to obtain global image features corresponding to each of the image scan sequence combinations; inputting each of the global image features into a pre-trained image recognition model to obtain feature recognition results corresponding to each of the image scan sequence combinations; wherein the training process of the image recognition model is designed to iteratively train based on a first loss of a full sequence combination in the MR image scan sequence combination set and a true label and a second loss of a non-full sequence combination and its neighborhood object; comprehensively analyzing each of the feature recognition results, and taking the image scan sequence combination corresponding to the analysis result with the highest comprehensive score as the optimal scan sequence combination.

2. The self-supervised knowledge transfer based multi-modality MR scan optimization method of claim 1, wherein, The method comprises: mask rule definition processing of sequence selection logic data of each of the multi-modal 3D image sequences based on an interpretable position coding technology to generate a plurality of mask vectors; arranging and combining each of the multi-modal 3D image sequences according to the mask vectors to obtain an MR image scan sequence combination set composed of a plurality of image scan sequence combinations; wherein the MR image scan sequence combination set includes a full sequence combination composed of all multi-modal 3D image sequences and a non-full sequence combination composed of part of the multi-modal 3D image sequences.

3. The self-supervised knowledge transfer based multi-modality MR scan optimization method of claim 1, wherein, The method comprises: performing deep feature mining processing on each of the multi-modal 3D image sequences to obtain deep image features corresponding to each of the multi-modal 3D image sequences; wherein the deep feature mining processing is used to capture features related to key differences of target images in the multi-modal 3D image sequences; performing dimension mapping and condensing processing on the deep image features to obtain high-dimensional feature representations corresponding to each of the multi-modal 3D image sequences.

4. The self-supervised knowledge transfer based multi-modality MR scan optimization method of claim 1, wherein, The method comprises: establishing a mask position dictionary corresponding to the MR image scan sequence combination set based on an interpretable position coding technology, wherein each image scan sequence combination corresponds to a mask vector in the mask position dictionary, and each component of the mask vector corresponds to a multi-modal 3D image sequence, and a component value of 1 indicates that the multi-modal 3D image sequence belongs to the current image scan sequence combination, and a component value of 0 indicates that the multi-modal 3D image sequence does not belong to the current image scan sequence combination; matching the mask position dictionary with a target mask vector corresponding to any image scan sequence combination in the MR image scan sequence combination set, and according to the component values of the target mask vector, filtering high-dimensional feature representations corresponding to component values of 1 from all multi-modal 3D image sequence high-dimensional feature representations as to-be-fused features; performing weighted accumulation processing on the filtered to-be-fused features to obtain global image features corresponding to the image scan sequence combination, wherein the weight of each to-be-fused feature is the component value of the target mask vector corresponding to the to-be-fused feature.

5. The self-supervised knowledge transfer based multi-modality MR scan optimization method of claim 1, wherein, Before inputting each global image feature into a pre-trained image recognition model, further comprising: obtaining an original multi-modal MR image sequence set and its corresponding real labels; performing feature extraction on each 3D image sequence in the original multi-modal MR image sequence set to obtain historical high-dimensional feature representations; performing first-stage joint training on a pre-constructed initial target image feature prediction model based on the historical high-dimensional feature representations and the image real feature labels to obtain a first image recognition model; performing permutation and combination on the multi-modal MR image sequence set based on an interpretable position coding technology to obtain full-sequence combination image features and non-full-sequence combination image features; performing second-stage joint training on the first image recognition model based on the full-sequence combination image features and the non-full-sequence combination image features to obtain a trained image recognition model.

6. The self-supervised knowledge transfer based multi-modality MR scan optimization method of claim 5, wherein, The second-stage joint training on the first image recognition model based on the full-sequence combination image features and the non-full-sequence combination image features to obtain a trained image recognition model comprises: inputting the full-sequence combination image features into the first image recognition model, outputting a first recognition result corresponding to the full-sequence combination, and calculating a cross-entropy loss between the first recognition result and the real label as a teacher loss; determining a learning object within a Hamming distance neighborhood of each non-full-sequence combination image feature, wherein the learning object is an MR sequence combination obtained by adding one 3D image sequence feature to the non-full-sequence combination image feature and containing all 3D image sequence features of the non-full-sequence combination image feature; respectively inputting the non-full-sequence combination image features and their corresponding learning objects into the first image recognition model to obtain a second recognition result of the non-full-sequence combination image features and a third recognition result of the corresponding learning objects, and calculating a mean square error loss between the second recognition result and the third recognition result as a student loss; iteratively optimizing the first image recognition model based on the teacher loss and the student loss to obtain a trained image recognition model.

7. The self-supervised knowledge transfer based multi-modality MR scan optimization method of claim 1, wherein, The comprehensive analysis of each feature recognition result is performed, and the image scanning sequence combination corresponding to the analysis result with the highest comprehensive score is taken as the optimal scanning sequence combination, comprising: Taking the first identification result of the full sequence image scanning combination as a benchmark, non-full sequence combinations with no statistical difference in identification performance from the first identification result are screened out from the feature identification results of each non-full sequence image scanning combination to form an initial candidate combination set; The safety risk level of each initial candidate combination in the initial candidate combination set is determined, wherein the safety risk level is determined based on the number of MR enhancement sequences in each initial candidate combination, and the MR enhancement sequence includes an MR scanning sequence using a contrast agent; The initial candidate combination set is subjected to secondary screening based on the safety risk level to obtain an optimal scanning sequence combination.

8. A multi-modality MR scan optimization system based on self-supervised knowledge transfer, characterized in that, Comprise: An acquisition module is configured to acquire a plurality of multi-modal 3D image sequences in response to an image processing signal; A combination module is configured to arrange and combine each of the multi-modal 3D image sequences based on an interpretable position coding technology to obtain a set of MR image scanning sequence combinations, wherein each image scanning sequence combination in the set of MR image scanning sequence combinations is different; An extraction module is configured to perform feature extraction processing on the multi-modal 3D image sequences to obtain high-dimensional feature representations corresponding to each of the multi-modal 3D image sequences, Wherein the feature extraction is designed to extract spatial features and semantic features associated with image recognition; A fusion module is configured to fuse the high-dimensional feature representations with the set of MR image scanning sequence combinations to obtain global image features corresponding to each of the image scanning sequence combinations; A prediction module is configured to input each of the global image features into a pre-trained image recognition model to obtain feature recognition results corresponding to each of the image scanning sequence combinations, wherein the training process of the image recognition model is designed to iteratively train based on a first loss of a full sequence combination in the set of MR image scanning sequence combinations and a true label and a second loss of a non-full sequence combination and its neighborhood object; An analysis module is configured to perform comprehensive analysis on each of the feature recognition results, and take the image scanning sequence combination corresponding to the analysis result with the highest comprehensive score as the optimal scanning sequence combination.

9. A multi-modality MR scan optimization device based on self-supervised knowledge transfer, characterized in that, The computer readable storage medium stores a computer program, wherein when the device where the computer readable storage medium is located executes the computer program, the self-supervised knowledge transfer based multi-modal MR scanning optimization method as claimed in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, wherein when the device where the computer readable storage medium is located executes the computer program, the self-supervised knowledge transfer based multi-modal MR scanning optimization method as claimed in any one of claims 1 to 7 is implemented.

Images