Multi-modal medical image fusion navigation management system

Abstract

The invention relates to a multi-modal medical image fusion navigation management system. The system comprises an image processing module which is used for acquiring a medical image data set and performing independent optimization processing on the medical image data set to generate a first image set; and the image fusion module is used for extracting multi-scale key points based on the first image set, generating a second feature set, performing cross-modal registration on the second feature set, calculating to obtain a third mapping set, and performing reconstruction to obtain an accurate image fusion image when the registration error of the third mapping set exceeds a preset threshold value. And the visual diagnosis and treatment module is used for extracting dynamic organization boundary features and a multi-modal fusion result based on the image fusion image, performing joint optimization by using a generative adversarial network and a reinforcement learning strategy, and finally generating a visual diagnosis and treatment path plan. The system provides comprehensive and clear comprehensive image information of diseased regions and surrounding tissues, assists in accurate diagnosis of illness states, and greatly improves scientificity and operability of diagnosis and treatment schemes.

Description

Technical Field

[0001] The present invention belongs to the field of medical imaging technology, and in particular relates to a multimodal medical image fusion navigation management system. Background Art

[0002] With the advancement of medical imaging technology, multimodal medical image fusion navigation and management technology has emerged. Single-modality medical images, such as CT (computed tomography), can clearly visualize the anatomical structure of bones and some tissues, but their resolution of soft tissue detail is limited. MRI (magnetic resonance imaging) has significant advantages in visualizing soft tissues, but struggles to effectively detect metabolic activity. PET (positron emission tomography) can accurately reflect human metabolic function, but its anatomical imaging is relatively ambiguous. These limitations make it difficult for clinicians to obtain complete and accurate information about a patient's condition based on a single image. To overcome this dilemma, multimodal medical image fusion technology has emerged, integrating the strengths of different imaging modalities to provide doctors with richer and more accurate diagnostic evidence. However, existing multimodal image fusion systems still cannot fully meet the complex and ever-changing clinical needs in terms of image processing efficiency, fusion accuracy, and the convenience and intelligence of navigation and management functions. Summary of the Invention

[0003] Based on this, it is necessary to address the above technical issues and provide a multimodal medical image fusion navigation management system that can provide more comprehensive and clear comprehensive imaging information of the lesion site and surrounding tissues, help accurately diagnose the disease, and improve the quality of diagnosis and treatment.

[0004] In a first aspect, the present application provides a multimodal medical image fusion navigation management system, comprising:

[0005] The image processing module is used to obtain a medical image data set; it is also used to independently optimize the medical image data set to obtain a first image set; the medical image data set includes medical image data of three modalities: CT, MRI, and PET.

[0006] The image fusion module is used to extract multi-scale key points based on the pixel distribution characteristics of the first image set and generate a second feature set in combination with the spatial transformation function; it is also used to perform cross-modal registration on the second feature set and obtain a third mapping set using the spatial transformation matrix calculation; if the registration error of the third mapping set exceeds a preset threshold, the multimodal image fusion algorithm based on deep learning is used to reconstruct the image fusion image.

[0007] The visual diagnosis and treatment module is used to extract dynamic tissue boundary features and multimodal fusion results based on image fusion, and use generative adversarial networks combined with reinforcement learning strategies for joint optimization to obtain visual diagnosis and treatment path planning.

[0008] In one embodiment, extracting multi-scale key points based on pixel distribution features of the first image set and generating a second feature set in combination with a spatial transformation function includes:

[0009] The pixel distribution features of the first image set are extracted using an adaptive histogram statistics method to obtain multi-scale key points.

[0010] The clustering algorithm is used to determine the input parameters of the spatial transformation of multi-scale key points and the formula is used to calculate to obtain the key point response value.

[0011] The spatial transformation function is executed according to the input parameters to perform geometric transformation, and the convolution operation is used to extract features to obtain the second feature.

[0012] If the point set analysis result of the second feature matches the key point set, the feature generation is optimized by adjusting the spatial transformation function to obtain the optimized second feature.

[0013] The affine transformation parameters are calculated based on the key point response values, and the affine transformation parameters are input into the adjusted spatial transformation function to obtain the feature matching matrix.

[0014] The optimized second features are aligned according to the feature matching matrix to obtain a second feature set.

[0015] In one embodiment, a clustering algorithm is used to determine the input parameters of the spatial transformation of the multi-scale key points and a formula is used to calculate the key point response value, including:

[0016] The key point response value is calculated using the following formula:

[0017]

[0018] Among them, R(P i ) represents the key point response value, P i Represents the key point, set the key point P i The coordinates are (xi,yi), N(P i ) represents the set of pixels in the neighborhood, and the pixel point q∈N(P i ), the coordinates of pixel point q are (x q ,y q ), I(Q) represents pixel intensity, I(P i ) represents the key point P i Own pixel intensity, s i represents the scale, α and β represent weight coefficients, which are used to balance the influence of neighborhood pixel intensity difference and scale information on the response value, and σ represents the standard deviation of the Gaussian kernel, which controls the attenuation degree of the contribution of the neighborhood pixel points to the key point response value. It can generally be set to σ = γ * s i , γ represents a constant.

[0019] In one embodiment, cross-modal registration is performed on the second feature set, and a third mapping set is obtained by calculating a spatial transformation matrix, including:

[0020] A feature descriptor is constructed for the second feature set to obtain a feature description vector of each modal feature.

[0021] Feature matching is performed based on the feature description vector to obtain an initial set of matching point pairs.

[0022] The random sampling consensus algorithm is used to screen the initial matching point pair set to obtain reliable matching point pairs.

[0023] The initial spatial transformation matrix is estimated based on reliable matching point pairs to obtain the initial spatial transformation matrix.

[0024] The initial spatial transformation matrix is iteratively optimized to obtain the optimized spatial transformation matrix.

[0025] Based on the optimized spatial transformation matrix, each modal feature in the second feature set is uniformly mapped to the same spatial coordinate system to obtain a third mapping set.

[0026] In one embodiment, dynamic tissue boundary features and multimodal fusion results are extracted based on image fusion, and a generative adversarial network combined with a reinforcement learning strategy are used for joint optimization to obtain a visual diagnosis and treatment path planning, including:

[0027] The dynamic tissue boundary features in the image fusion image are obtained to generate dynamic feature sequence data.

[0028] The dynamic feature sequence data is input into the generative adversarial network, and adversarial training is used to output the multimodal fusion feature tensor.

[0029] The multimodal fusion feature tensor is input into the policy network to obtain the feature optimization policy gradient parameters.

[0030] Based on the policy gradient parameter update, the adversarial loss function of the generative adversarial network is generated to obtain the optimized three-dimensional diagnosis and treatment pathway map.

[0031] The key topological nodes in the three-dimensional diagnosis and treatment pathway map are extracted to construct a hierarchical path decision tree.

[0032] The hierarchical path decision tree is aligned with the dynamic feature sequence data in time and space, and a visual diagnosis and treatment path planning is generated after iterative correction by the reinforcement learning strategy.

[0033] In one embodiment, the multimodal fusion feature tensor is input into the policy network to obtain the feature optimization policy gradient parameters, including:

[0034] Perform feature dimensionality reduction on the multimodal fusion feature tensor to obtain a low-dimensional feature representation.

[0035] The policy network is forward propagated based on the low-dimensional feature representation to obtain the intermediate policy output.

[0036] The gradient of the intermediate strategy output is calculated according to the preset optimization objective and loss function to obtain the feature optimization strategy gradient parameters.

[0037]

[0038] L(θ)=-J(y)

[0039] y=fθ(x)

[0040] in, Represents the feature optimization policy gradient parameter, assumes the policy network is fθ, the input low-dimensional feature is x, y represents the intermediate policy output, the optimization objective function is J(y), and L(θ) represents the loss function.

[0041] In a second aspect, the present application also provides a multimodal medical image fusion navigation management method, the method comprising:

[0042] A medical imaging dataset is obtained; and the medical imaging dataset is independently optimized to obtain a first image set; the medical imaging dataset includes medical imaging data of three modalities: CT, MRI, and PET.

[0043] Multi-scale key points are extracted from the pixel distribution features of the first image set, and a second feature set is generated by combining the spatial transformation function. The second feature set is cross-modally registered, and a third mapping set is obtained by calculating the spatial transformation matrix. If the registration error of the third mapping set exceeds a preset threshold, a multimodal image fusion algorithm based on deep learning is used to reconstruct the image fusion image.

[0044] Based on image fusion, dynamic tissue boundary features and multimodal fusion results are extracted, and joint optimization is performed using a generative adversarial network combined with a reinforcement learning strategy to obtain a visual diagnosis and treatment path planning.

[0045] In a third aspect, the present application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the above system when executing the computer program.

[0046] In a fourth aspect, the present application also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the above system is implemented.

[0047] The multimodal medical image fusion navigation and management system mentioned above includes three key modules: image processing, image fusion, and visual diagnosis and treatment. The image processing module is responsible for acquiring medical image datasets covering three modalities: CT, MRI, and PET, and independently optimizing them to generate a first image set. The image fusion module extracts multi-scale key points based on the pixel distribution characteristics of the first image set, combines them with a spatial transformation function to generate a second feature set, then performs cross-modal registration on the second feature set and calculates a third mapping set using a spatial transformation matrix. Once the registration error of the third mapping set exceeds a preset threshold, it is reconstructed using a deep learning multimodal image fusion algorithm to obtain an accurate image fusion image. The visual diagnosis and treatment module extracts dynamic tissue boundary features and multimodal fusion results from the image fusion image, and jointly optimizes them using a generative adversarial network and reinforcement learning strategy to ultimately generate a visual diagnosis and treatment path plan. The above system provides comprehensive and clear integrated imaging information of the lesion site and surrounding tissues, assisting in accurate diagnosis of the condition, greatly improving the scientific nature and operability of treatment plans, and providing clinicians with efficient and accurate decision support, thereby improving patient diagnosis and treatment outcomes. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0049] Figure 1 A structural block diagram of a multimodal medical image fusion navigation management system provided by an embodiment of the present invention;

[0050] Figure 2 A flowchart of extracting multi-scale key points based on pixel distribution features of a first image set and generating a second feature set in combination with a spatial transformation function provided by an embodiment of the present invention;

[0051] Figure 3 A flowchart of performing cross-modal registration on the second feature set and obtaining a third mapping set by calculating a spatial transformation matrix provided by an embodiment of the present invention;

[0052] Figure 4 The embodiment of the present invention provides a flowchart for visualizing diagnosis and treatment path planning by extracting dynamic tissue boundary features and multimodal fusion results based on image fusion, and using a generative adversarial network combined with a reinforcement learning strategy for joint optimization;

[0053] Figure 5 The present invention provides a flowchart of a multimodal medical image fusion navigation management method. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0055] First, the implementation environment of the embodiment of the present application is described. By way of example, the implementation environment includes an image acquisition device, a data processing system, and a display and interaction device.

[0056] In a multimodal medical image fusion navigation management system, the image acquisition device transmits the collected original medical image data to the data processing system in a wired or wireless manner through a network switch. The high-performance computer in the system uses its multi-core processor, large-capacity memory and professional graphics processing unit to quickly process and analyze the data, covering operations such as image reconstruction and fusion algorithm calculations. After the processed data is temporarily stored on the storage server, the fused image data is again transmitted to the display and interactive device with the help of the network switch. This device includes a high-resolution display, mouse, keyboard, handwriting tablet, etc. The high-resolution display is used to clearly present the image, and the doctor uses input devices such as mouse, keyboard, handwriting tablet, etc. to input commands, mark and perform other operations based on the displayed image, so as to achieve efficient and stable data interaction and smooth collaboration between the key components in the system.

[0057] In combination with the above implementation environment, the application scenarios of the embodiments of the present application are explained.

[0058] The embodiment of the present application provides a multimodal medical image fusion navigation management system that integrates data obtained from a variety of image acquisition devices such as CT, MRI, PET and ultrasound, and uses advanced image fusion algorithms to accurately align and fuse multi-source images in the data processing system. The system relies on the powerful computing power of high-performance computers to achieve rapid processing and analysis of complex data, presents fused images through high-resolution displays, and provides doctors with intuitive and comprehensive medical imaging information with interactive devices such as mice and keyboards. It significantly improves diagnostic accuracy and treatment accuracy, and strongly promotes the development and progress of modern medical diagnosis and treatment technologies. For example, the multimodal medical image fusion navigation management system provided by the embodiment of the present application can be applied to at least one of the following scenarios, including but not limited to.

[0059] First, this multimodal medical image fusion navigation management system is applied to surgical navigation. During the navigation process of complex surgical operations, before the operation, CT, MRI and other imaging acquisition equipment obtain detailed anatomical structure imaging data of the patient, and PET equipment provides metabolic information of the lesion site. The data processing system uses advanced algorithms to quickly fuse and analyze these multi-source data to generate an accurate three-dimensional model. During the operation, the display and interactive equipment presents the fused image in real time. By operating the input device, the surgeon can intuitively view the spatial relationship between the lesion and the surrounding blood vessels and nerves, and perform precise operations based on the navigation information to ensure the safety and efficiency of the operation, significantly improve the success rate of the operation, and reduce the risk of complications.

[0060] Second, a multimodal medical image fusion navigation and management system is being applied to disease diagnosis. In this area, the multimodal medical image fusion navigation and management system helps doctors make more accurate judgments. Image acquisition equipment captures patient images from various angles, such as ultrasound equipment for real-time soft tissue imaging, complementing the shortcomings of other equipment. The data processing system integrates these images, extracts hidden information, and enhances the identification of lesion features. Using high-resolution displays and interactive devices, doctors observe the fused images and conduct detailed analysis of lesions. This allows for the early detection of conditions such as tiny brain tumors and early cardiovascular lesions, buying valuable time for timely treatment and improving the accuracy and timeliness of disease diagnosis.

[0061] Third, a multimodal medical image fusion navigation management system is applied to telemedicine. In telemedicine scenarios, the multimodal medical image fusion navigation management system breaks geographical restrictions. Primary medical institutions use image acquisition equipment to obtain multimodal image data of patients and transmit it to the data processing center via the network. After the data processing system professionally processes and fuses the data, it allows remote experts to clearly view the details of the patient's images through display and interactive devices. Experts can use interactive devices to mark and measure, communicate with primary doctors in real time, and guide diagnosis and treatment plan formulation. This application realizes the sharing of high-quality medical resources, improves the level of medical services in remote areas, and allows more patients to benefit from advanced medical technology.

[0062] In one embodiment, Figure 1 As shown, the present application provides a multimodal medical image fusion navigation management system, which may include:

[0063] The image processing module 101 is used to obtain a medical image data set; it is also used to independently optimize the medical image data set to obtain a first image set; the medical image data set includes medical image data of three modalities: CT, MRI, and PET.

[0064] This module connects with data from various medical imaging acquisition devices to comprehensively and accurately acquire medical imaging datasets including CT (computed tomography), MRI (magnetic resonance imaging), and PET (positron emission tomography). These different modalities of imaging data reflect the internal structure and function of the human body from their own unique perspectives. However, their original data may have problems such as noise interference, poor contrast, and resolution differences. Therefore, the image processing module independently optimizes the acquired medical imaging datasets, using advanced image enhancement algorithms, noise reduction techniques, and normalization methods to sharpen bone tissue edges and remove scanning artifacts in CT images, improve the clarity of soft tissue details in MRI images, and correct radioactive distribution deviations in PET images, thereby obtaining a first image set with better quality and consistency.

[0065] The image fusion module 102 is used to extract multi-scale key points based on the pixel distribution characteristics of the first image set and generate a second feature set in combination with a spatial transformation function; it is also used to perform cross-modal registration on the second feature set and obtain a third mapping set using a spatial transformation matrix; if the registration error of the third mapping set exceeds a preset threshold, a multimodal image fusion algorithm based on deep learning is used to reconstruct the image to obtain an image fusion image.

[0066] First, based on the first image set, the pixel distribution characteristics are deeply analyzed, and cutting-edge algorithms such as adaptive histogram statistics are used to accurately extract multi-scale key points. These key points contain rich image structure and texture information and are key elements for subsequent processing. The module combines spatial transformation functions and generates a second feature set that can comprehensively reflect image features at different scales by performing geometric transformations such as translation, rotation, and scaling on the key points. To achieve precise alignment of image features of different modalities, the module performs cross-modal registration operations on the second feature set, using feature matching algorithms and random sampling consensus algorithms to screen out reliable matching point pairs from different modal features, and then estimate the initial spatial transformation matrix. The accurate spatial transformation matrix is obtained through iterative optimization, and the third mapping set is calculated based on this, which uniformly maps the features of different modalities to the same spatial coordinate system. If the registration error of the third mapping set exceeds the pre-set threshold, indicating that the registration accuracy does not meet the ideal standard, the module will then activate the deep learning multimodal image fusion algorithm. With the help of the powerful feature learning and data fitting capabilities of the deep neural network, the image is reconstructed, effectively integrating the advantages of each modality of imaging, and ultimately generating a high-quality image fusion image, providing a comprehensive and accurate imaging basis for subsequent visual diagnosis and treatment and precision medical decision-making.

[0067] The visualization diagnosis and treatment module 103 is used to extract dynamic tissue boundary features and multimodal fusion results based on image fusion images, and use generative adversarial networks combined with reinforcement learning strategies for joint optimization to obtain visualization diagnosis and treatment path planning.

[0068] This module deeply mines high-quality image fusion images, applying advanced algorithms such as edge detection and contour recognition to accurately extract dynamic tissue boundary features and capture information about tissue morphological changes under different physiological or pathological conditions. Furthermore, it comprehensively analyzes the multimodal fusion results, comprehensively considering the multi-dimensional information provided by the different imaging modalities, including anatomical structure and metabolic function. The module incorporates cutting-edge generative adversarial network technology, which comprises a generator and a discriminator. The generator strives to simulate a real-world treatment pathway, while the discriminator verifies the authenticity of the generated pathway, working in a mutually reinforcing and continuous optimization process. Incorporating a reinforcement learning strategy, a rational reward mechanism provides feedback based on the actual effectiveness of the generated pathway in guiding disease diagnosis and treatment, guiding the network to continuously adjust its strategy and achieve continuous optimization of the treatment pathway. After multiple rounds of iteration, a visual treatment pathway plan is generated, presented in clear graphics, charts, and annotations, providing clinicians with a clear and comprehensive reference for the treatment process, significantly improving the efficiency and accuracy of medical decision-making and facilitating the efficient implementation of precision medicine.

[0069] The multimodal medical image fusion navigation management method includes three key modules: image processing, image fusion, and visual diagnosis and treatment. The image processing module is responsible for acquiring medical image datasets covering three modalities: CT, MRI, and PET, and independently optimizing them to generate a first image set. The image fusion module extracts multi-scale key points based on the pixel distribution features of the first image set, combines them with a spatial transformation function to generate a second feature set, then performs cross-modal registration on the second feature set and calculates a third mapping set using a spatial transformation matrix. Once the registration error of the third mapping set exceeds a preset threshold, it is reconstructed using a deep learning multimodal image fusion algorithm to obtain an accurate image fusion image. The visual diagnosis and treatment module extracts dynamic tissue boundary features and multimodal fusion results from the image fusion image, and jointly optimizes them using a generative adversarial network and reinforcement learning strategy to ultimately generate a visual diagnosis and treatment path plan. The system provides comprehensive and clear image information of the lesion site and surrounding tissues, facilitating accurate diagnosis, significantly improving the scientific nature and operability of treatment plans, and providing clinicians with efficient and accurate decision support, thereby improving patient treatment outcomes.

[0070] In one embodiment, Figure 2 As shown, extracting multi-scale key points based on the pixel distribution features of the first image set and generating a second feature set in combination with a spatial transformation function may include the following steps:

[0071] Step S201 : extracting multi-scale key points based on pixel distribution features of the first image set using an adaptive histogram statistics method.

[0072] Step S202: Using a clustering algorithm to determine the input parameters of the spatial transformation of the multi-scale key points and using a formula to perform calculations to obtain key point response values.

[0073] Step S203: executing a spatial transformation function to perform geometric transformation according to the input parameters, and extracting features using a convolution operation to obtain a second feature.

[0074] Step S204: If the point set analysis result of the second feature matches the key point set, feature generation is optimized by adjusting the spatial transformation function to obtain an optimized second feature.

[0075] Step S205 : Calculate affine transformation parameters based on the key point response values, and input the affine transformation parameters into the adjusted spatial transformation function to obtain a feature matching matrix.

[0076] Step S206 , aligning the optimized second features according to the feature matching matrix to obtain a second feature set.

[0077] First, based on the pixel distribution characteristics of the first image set, an adaptive histogram statistics method is used to accurately extract multi-scale keypoints by dynamically adjusting the histogram statistical interval. Subsequently, a clustering algorithm is applied to these multi-scale keypoints. During the clustering process, the input parameters of the spatial transformation are simultaneously determined and calculated using a specialized formula to obtain the keypoint response values. Next, based on the determined input parameters, a spatial transformation function is applied to the image for geometric transformation. Convolution operations are then used to extract features, thereby generating the second feature. If the point set analysis results of the second feature match the keypoint set, the spatial transformation function is adjusted to optimize feature generation, thereby generating the optimized second feature. Next, affine transformation parameters are calculated based on the keypoint response values and input into the adjusted spatial transformation function to obtain a feature matching matrix. Finally, based on this feature matching matrix, feature alignment is performed on the optimized second features to generate the second feature set.

[0078] The adaptive histogram statistics method combined with clustering algorithms and other operations can fully tap into the potential features of the first image set, improve the accuracy and comprehensiveness of key point extraction, and provide rich and high-quality basic data for subsequent processing. Through multiple rounds of optimization, such as adjusting the spatial transformation function based on the point set matching results and calculating the affine transformation parameters using the key point response values, the quality and consistency of the second feature set can be effectively improved. This enables the features of images of different modalities to be better integrated and aligned under a unified framework, providing a more reliable feature foundation for subsequent key links such as cross-modal registration and image fusion, helping to improve the accuracy of medical image analysis, assisting doctors in more accurately diagnosing diseases, formulating more scientific treatment plans, and improving the quality and efficiency of medical services.

[0079] In one embodiment, determining the input parameters of the spatial transformation using a clustering algorithm for multi-scale key points and calculating using a formula to obtain key point response values may include the following steps:

[0080] The key point response value is calculated using the following formula:

[0081]

[0082] Among them, R(P i ) represents the key point response value, P i Represents the key point, set the key point P i The coordinates are (xi,yi), N(P i ) represents the set of pixels in the neighborhood, and the pixel point q∈N(P i ), the coordinates of pixel point q are (x q ,y q ), I(Q) represents pixel intensity, I(P i ) represents the key point P i Own pixel intensity, s i represents the scale, α and β represent weight coefficients, which are used to balance the influence of neighborhood pixel intensity difference and scale information on the response value, and σ represents the standard deviation of the Gaussian kernel, which controls the attenuation degree of the contribution of the neighborhood pixel points to the key point response value. It can generally be set to σ = γ * s i , γ represents a constant.

[0083] This embodiment can comprehensively and accurately reflect the importance and stability of key points in the image by comprehensively considering the pixel intensity of the key points themselves, the difference in pixel intensity in the neighborhood, and the scale information. It effectively balances the influence of the difference in neighborhood pixel intensity and scale information on the response value, and enhances the adaptability and universality of the formula. The Gaussian kernel standard deviation is set according to the scale information, which can reasonably control the attenuation degree of the contribution of the pixel points in the neighborhood to the key point response value, so that the characteristics of the key points can be accurately captured at different scales. The key point response value calculated by this formula provides a reliable and representative basis for subsequent operations such as determining spatial transformation input parameters and calculating affine transformation parameters, significantly improving the accuracy of feature extraction and analysis in the multimodal medical image fusion processing process, helping to improve image fusion accuracy, and providing doctors with more accurate basis for disease judgment, thereby improving the quality of medical diagnosis and treatment.

[0084] In one embodiment, Figure 3 As shown, performing cross-modal registration on the second feature set and obtaining the third mapping set by calculating the spatial transformation matrix may include the following steps:

[0085] Step S301: construct a feature descriptor for the second feature set to obtain a feature description vector for each modal feature.

[0086] Step S302: Perform feature matching based on the feature description vector to obtain an initial matching point pair set.

[0087] Step S303: Using a random sampling consensus algorithm, the initial matching point pair set is screened to obtain reliable matching point pairs.

[0088] Step S304 : Estimating an initial spatial transformation matrix based on the reliable matching point pairs to obtain an initial spatial transformation matrix.

[0089] Step S305: iteratively optimize the initial spatial transformation matrix to obtain an optimized spatial transformation matrix.

[0090] Step S306 : Based on the optimized spatial transformation matrix, all modal features in the second feature set are uniformly mapped to the same spatial coordinate system to obtain a third mapping set.

[0091] First, a feature descriptor is constructed for the second feature set. An algorithm is used to generate a corresponding feature description vector for each modal feature, converting the complex feature information into a vector form that facilitates comparison and analysis. Feature matching is then performed based on these feature description vectors. By measuring the similarity between the vectors, an initial set of matching point pairs is obtained, preliminarily establishing the association between the different modal features. A random sampling consensus algorithm is then used to rigorously screen this initial set of matching point pairs, removing any potentially mismatched pairs to obtain reliable matching point pairs and ensure the accuracy and reliability of the matching results. Based on these reliable matching point pairs, an appropriate mathematical model is used to estimate the initial spatial transformation matrix, laying the foundation for subsequent spatial transformations. This initial spatial transformation matrix is then iteratively optimized, with the matrix parameters continuously adjusted to more accurately reflect the spatial relationships between the different modal features. Ultimately, the optimized spatial transformation matrix is obtained. Finally, using this optimized spatial transformation matrix, the modal features in the second feature set are uniformly mapped to the same spatial coordinate system, completing the spatial alignment of the different modal features and generating the third mapping set.

[0092] The feature descriptor construction and feature matching steps can effectively explore the similarities between different modal features, provide initial correspondences for cross-modal registration, and greatly improve the efficiency and accuracy of registration. The application of the random sampling consensus algorithm effectively eliminates the interference of mismatched point pairs, improves the reliability of matching results, and ensures the accuracy of subsequent spatial transformation matrix estimation. The estimation of the initial spatial transformation matrix and subsequent iterative optimization can gradually optimize the alignment of different modal features in space, so that the final generated third mapping set can more accurately reflect the true distribution of each modal feature in the same spatial coordinate system. Providing doctors with more accurate and comprehensive imaging information will help improve the accuracy of disease diagnosis and the scientific nature of treatment plan formulation.

[0093] In one embodiment, Figure 4 As shown in the figure, based on the image fusion image, dynamic tissue boundary features and multimodal fusion results are extracted, and the generative adversarial network is combined with the reinforcement learning strategy for joint optimization to obtain a visual diagnosis and treatment path planning, which can include the following steps:

[0094] Step S401: Acquire dynamic tissue boundary features in the image fusion image and generate dynamic feature sequence data.

[0095] In step S402, the dynamic feature sequence data is input into a generative adversarial network, and adversarial training is used to output a multimodal fusion feature tensor.

[0096] Step S403: Input the multimodal fusion feature tensor into the strategy network to obtain feature optimization strategy gradient parameters.

[0097] Step S404: Update the adversarial loss function of the adversarial network based on the policy gradient parameters to obtain an optimized three-dimensional diagnosis and treatment pathway map.

[0098] Step S405: extract key topological nodes in the three-dimensional diagnosis and treatment pathway map to construct a hierarchical pathway decision tree.

[0099] Step S406: align the hierarchical path decision tree with the dynamic feature sequence data in time and space, and generate a visual diagnosis and treatment path plan after iterative correction using the reinforcement learning strategy.

[0100] First, advanced edge detection and tracking algorithms are applied to accurately capture dynamic tissue boundary features from high-quality image fusion images. These dynamic feature sequence data are then organized and generated according to the order of temporal or physiological state changes. Next, this dynamic feature sequence data is fed into a carefully constructed generative adversarial network (GAN). The GAN comprises a generator and a discriminator. Through continuous adversarial training, the generator's output is continuously optimized, resulting in a multimodal fusion feature tensor that integrates the combined features of multimodal imaging and dynamic tissue boundaries. This multimodal fusion feature tensor is then fed into a policy network. Based on a pre-set optimization objective, the policy network analyzes and calculates the input data to determine the feature optimization policy gradient parameters. Based on these policy gradient parameters, the GAN's adversarial loss function is updated. By adjusting the loss function, the GAN is guided to generate results that better meet practical needs, resulting in an optimized three-dimensional diagnosis and treatment pathway map that initially represents the three-dimensional architecture of the diagnosis and treatment process. Subsequently, the three-dimensional treatment pathway map was deeply mined to extract key topological nodes. Based on the logical relationships between the nodes, a hierarchical pathway decision tree was constructed, presenting the decision branches of the treatment pathway in a clear hierarchical structure. Finally, the hierarchical pathway decision tree was spatially and temporally aligned with the dynamic feature sequence data to ensure that the pathway planning matched the dynamic changes of the organization in both time and space. Reinforcement learning strategies were then used to iteratively correct the pathway, continuously optimizing the pathway based on actual feedback. Ultimately, a visual treatment pathway plan was generated and presented to clinicians in an intuitive and easy-to-understand format.

[0101] This embodiment accurately extracts dynamic tissue boundary features from image fusion images and generates sequence data, which can reflect the changes in diseased tissue in real time and provide dynamic information support for diagnosis and treatment decisions. The collaborative work of the generative adversarial network and the policy network significantly improves the accuracy and rationality of diagnosis and treatment path planning by continuously optimizing the feature tensor and loss function. The construction of a hierarchical path decision tree provides doctors with a clear decision-making framework, which helps to formulate diagnosis and treatment plans quickly and accurately. The application of spatiotemporal alignment and reinforcement learning strategies enables diagnosis and treatment path planning to closely fit the patient's actual physiological state, enhancing the practicality and reliability of the plan. The final generated visual diagnosis and treatment path planning greatly improves the comprehensibility of medical information.

[0102] In one embodiment, inputting the multimodal fusion feature tensor into the policy network to obtain feature optimization policy gradient parameters may include the following steps:

[0103] Step S501: perform feature dimensionality reduction processing on the multimodal fusion feature tensor to obtain a low-dimensional feature representation.

[0104] Step S502: Perform forward propagation of the policy network based on the low-dimensional feature representation to obtain an intermediate policy output.

[0105] Step S503: Perform gradient calculation on the intermediate strategy output according to the preset optimization objective and loss function to obtain the feature optimization strategy gradient parameter.

[0106]

[0107] L(θ)=-J(y)

[0108] y=fθ(x)

[0109] in, Represents the feature optimization policy gradient parameter, assumes the policy network is fθ, the input low-dimensional feature is x, y represents the intermediate policy output, the optimization objective function is J(y), and L(θ) represents the loss function.

[0110] Feature dimensionality reduction processing significantly improves the efficiency of subsequent calculations, reduces the consumption of computing resources, and avoids the dimensionality disaster problem caused by high-dimensional data, enabling the policy network to process data more efficiently. The forward propagation of the policy network based on low-dimensional feature representation lays the foundation for generating reasonable intermediate policy outputs and ensures that policy decisions are based on valid data features. The gradient calculation based on the preset optimization objectives and loss function provides a clear direction for the parameter optimization of the policy network. By continuously adjusting the parameters of the policy network, the policy network can better adapt to the characteristics of multimodal medical image fusion data, improve the accuracy and reliability of the system's decision-making on key tasks such as diagnosis and treatment path planning, and thus provide clinicians with more scientific and accurate decision support, helping to improve the quality of medical services.

[0111] In one embodiment, Figure 5 As shown, the present application also provides a multimodal medical image fusion navigation management method, the method comprising:

[0112] Step S601 , obtaining a medical image data set; performing independent optimization processing on the medical image data set to obtain a first image set; the medical image data set includes medical image data of three modalities: CT, MRI, and PET.

[0113] In step S602, multi-scale key points are extracted from the pixel distribution features of the first image set, and a second feature set is generated by combining the spatial transformation function; the second feature set is cross-modally registered, and a third mapping set is obtained by calculating the spatial transformation matrix; if the registration error of the third mapping set exceeds a preset threshold, a multi-modal image fusion algorithm based on deep learning is used to reconstruct the image to obtain an image fusion image.

[0114] Step S603: extract dynamic tissue boundary features and multimodal fusion results based on the image fusion image, and use the generative adversarial network combined with the reinforcement learning strategy for joint optimization to obtain a visual diagnosis and treatment path planning.

[0115] The multimodal medical image fusion navigation management method first acquires a medical image dataset consisting of three modalities: CT, MRI, and PET. These datasets are independently optimized and processed, using advanced techniques such as image enhancement, noise reduction, and normalization to eliminate noise from the original data and enhance contrast and resolution, resulting in a high-quality first image set. Based on the pixel distribution characteristics of the first image set, multiscale keypoints are extracted using methods such as adaptive histogram statistics, and a spatial transformation function is combined to generate a second feature set. Cross-modal registration is then performed on the second feature set, using a feature matching algorithm and a random sampling consensus algorithm to select reliable matching point pairs. The spatial transformation matrix is estimated and optimized to calculate a third mapping set. If the registration error of the third mapping set exceeds a preset threshold, the images are reconstructed using a deep learning multimodal image fusion algorithm to obtain an accurate fused image. Finally, based on the fused image, dynamic tissue boundary features are extracted using algorithms such as edge detection and contour recognition. The multimodal fusion results are analyzed and fed into a generative adversarial network for joint optimization using a reinforcement learning strategy. Through adversarial training and strategy adjustment, a visual diagnosis and treatment pathway planning process is ultimately achieved. The above method provides comprehensive and clear imaging information of the lesion site and surrounding tissues, helps to accurately diagnose the disease, greatly improves the scientific nature and operability of the diagnosis and treatment plan, provides clinicians with efficient and accurate decision-making support, and thus improves the patient's diagnosis and treatment effect.

[0116] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0117] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the multimodal medical image fusion navigation management system as described above when executing the computer program.

[0118] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0119] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the partial description of the method embodiments. The device embodiments described above are merely illustrative, wherein the components described as separate parts may or may not be physically separated, and the parts displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the disclosed solution. A person of ordinary skill in the art can understand and implement it without expending creative work.

[0120] The above-described embodiments merely represent several implementation methods of the embodiments of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent application. It should be noted that a person skilled in the art may make various modifications and improvements without departing from the concept of the embodiments of the present application, and these modifications and improvements fall within the scope of protection of the embodiments of the present application.

Claims

1. A multimodal medical image fusion navigation management system, characterized in that: The system comprises: An image processing module is configured to acquire a medical image dataset and perform independent optimization processing on the medical image dataset to obtain a first image set; the medical image dataset includes medical image data of three modalities: CT, MRI, and PET; An image fusion module is configured to extract multi-scale key points based on the pixel distribution features of the first image set and generate a second feature set by combining the feature set with a spatial transformation function; further configured to perform cross-modal registration on the second feature set and calculate a third mapping set using a spatial transformation matrix; and if the registration error of the third mapping set exceeds a preset threshold, reconstruct the image using a multi-modal image fusion algorithm based on deep learning to obtain an image fusion image; The visualization diagnosis and treatment module is used to extract dynamic tissue boundary features and multimodal fusion results based on the image fusion image, and use the generative adversarial network combined with the reinforcement learning strategy for joint optimization to obtain visualization diagnosis and treatment path planning.

2. The system according to claim 1, wherein: The step of extracting multi-scale key points based on the pixel distribution features of the first image set and generating a second feature set in combination with a spatial transformation function includes: Extracting pixel distribution features of the first image set using an adaptive histogram statistics method to obtain multi-scale key points; Determining input parameters of spatial transformation for the multi-scale key points using a clustering algorithm and calculating using a formula to obtain key point response values; Executing a spatial transformation function to perform geometric transformation according to the input parameters, and extracting features using a convolution operation to obtain a second feature; If the point set analysis result of the second feature matches the key point set, optimizing feature generation by adjusting the spatial transformation function to obtain an optimized second feature; Calculating affine transformation parameters based on the key point response values, and inputting the affine transformation parameters into the adjusted spatial transformation function to obtain a feature matching matrix; Feature alignment is performed on the optimized second features according to the feature matching matrix to obtain a second feature set.

3. The system according to claim 2, characterized in that The method of determining the input parameters of the spatial transformation of the multi-scale key points using a clustering algorithm and calculating the key point response values using a formula includes: The key point response value is calculated using the following formula: Among them, R(P i ) represents the key point response value, P i Represents the key point, set the key point P i The coordinates of (x i ,y i ), N(P i ) represents the set of pixels in the neighborhood, and the pixel point q∈N(P i ), the coordinates of pixel point q are (x q ,y q ), I(Q) represents pixel intensity, I(P i ) represents the key point P i Own pixel intensity, s i represents the scale, α and β represent weight coefficients, which are used to balance the influence of neighborhood pixel intensity difference and scale information on the response value, and σ represents the standard deviation of the Gaussian kernel, which controls the attenuation degree of the contribution of the neighborhood pixel points to the key point response value. It can generally be set to σ = γ * s i , γ represents a constant.

4. The system according to claim 1, wherein: The performing cross-modal registration on the second feature set and obtaining a third mapping set by calculating a spatial transformation matrix includes: Constructing a feature descriptor for the second feature set to obtain a feature description vector for each modal feature; Perform feature matching based on the feature description vector to obtain an initial matching point pair set; Using a random sampling consensus algorithm to screen the initial matching point pair set to obtain reliable matching point pairs; Estimating an initial spatial transformation matrix based on the reliable matching point pairs to obtain an initial spatial transformation matrix; Iteratively optimizing the initial spatial transformation matrix to obtain an optimized spatial transformation matrix; Based on the optimized spatial transformation matrix, the modal features in the second feature set are uniformly mapped to the same spatial coordinate system to obtain a third mapping set.

5. The system according to claim 1, wherein: The method extracts dynamic tissue boundary features and multimodal fusion results based on the image fusion image, and uses a generative adversarial network combined with a reinforcement learning strategy for joint optimization to obtain a visual diagnosis and treatment path planning, including: Acquiring dynamic tissue boundary features in the image fusion image to generate dynamic feature sequence data; Inputting the dynamic feature sequence data into a generative adversarial network, and using adversarial training to output a multimodal fusion feature tensor; Inputting the multimodal fusion feature tensor into the strategy network to obtain feature optimization strategy gradient parameters; Updating the adversarial loss function of the generative adversarial network based on the policy gradient parameter to obtain an optimized three-dimensional diagnosis and treatment pathway map; Extracting key topological nodes from the three-dimensional diagnosis and treatment pathway map to construct a hierarchical pathway decision tree; The hierarchical path decision tree is spatially and temporally aligned with the dynamic feature sequence data, and a visual diagnosis and treatment path plan is generated after iterative correction by the reinforcement learning strategy.

6. The system according to claim 1, wherein: Inputting the multimodal fusion feature tensor into the strategy network to obtain feature optimization strategy gradient parameters includes: Performing feature dimensionality reduction processing on the multimodal fusion feature tensor to obtain a low-dimensional feature representation; Performing forward propagation of the policy network based on the low-dimensional feature representation to obtain an intermediate policy output; Performing gradient calculation on the intermediate strategy output according to the preset optimization objective and loss function to obtain the feature optimization strategy gradient parameter; L(θ)=-J(y) y=f θ (x) in, Represents the feature optimization policy gradient parameter, and the policy network is set to f θ , the input low-dimensional feature is x, y represents the intermediate strategy output, the optimization objective function is J(y), and L(θ) represents the loss function.

7. A multimodal medical image fusion navigation management method, characterized in that: The method comprises: Acquire a medical imaging dataset; perform independent optimization processing on the medical imaging dataset to obtain a first image set; the medical imaging dataset includes medical imaging data of three modalities: CT, MRI, and PET; Extracting multi-scale key points based on the pixel distribution features of the first image set and generating a second feature set in combination with a spatial transformation function; performing cross-modal registration on the second feature set and obtaining a third mapping set using a spatial transformation matrix; and reconstructing an image fusion image using a multimodal image fusion algorithm based on deep learning if the registration error of the third mapping set exceeds a preset threshold; Based on the image fusion, dynamic tissue boundary features and multimodal fusion results are extracted, and a generative adversarial network is combined with a reinforcement learning strategy for joint optimization to obtain a visual diagnosis and treatment path planning.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the system according to any one of claims 1 to 7 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the system according to any one of claims 1 to 7 are implemented.

Images