Image data processing method and device and storage medium

Abstract

The invention provides an image data processing method and device and a storage medium. According to the method, the medical image registration problem of the whole segment of the spine is decomposed into segmentation, pose transformation and splicing processing of a plurality of vertebral segments, so that the calculation complexity and the technical overhead of direct registration of the whole segment are remarkably reduced. According to the method, accurate segmentation of vertebral segments is realized by utilizing respective characteristics of a first modal medical image and a second modal medical image on bone imaging and soft tissue imaging; a pose transformation information set is determined based on a segmentation result, calculation of a transformation matrix is further optimized, and high efficiency and accuracy of the registration process are ensured; finally, the complete registration image is generated by performing weighted fusion splicing on the transformed segmented data set, so that the defect of overweight calculation burden caused by overall processing in the related technology is overcome, and the method has the characteristics of high real-time performance and wide applicability, and is particularly suitable for an application scene of real-time registration in an operation.

Description

Technical Field

[0001] This invention relates to the technical field of medical image processing, specifically to an image data processing method, device, and storage medium. Background Technology

[0002] Medical image registration is the process of geometrically aligning medical images acquired at different times, in different modalities, with different patient positions, or using different imaging devices. Its purpose is to align identical anatomical structures or lesion areas within a unified coordinate system, providing comprehensive information support for diagnosis, surgical planning, and treatment. This technology is widely used in multimodal image analysis, preoperative planning, intraoperative navigation, and disease progression monitoring.

[0003] In related technologies, there is a technical problem of excessive computational overhead in the application scenario of medical image registration of the entire spinal segment. Summary of the Invention

[0004] The purpose of this invention is to overcome the above-mentioned technical deficiencies and provide an image data processing method, device and storage medium to solve the technical problem of excessive computational overhead in the application scenario of medical image registration of the entire spinal segment in related technologies.

[0005] To achieve the above-mentioned technical objectives, the present invention adopts the following technical solution:

[0006] In a first aspect, the present invention provides an image data processing method, the method comprising:

[0007] Receive first modal medical image data and second modal medical image data representing a target vertebra; the first modal medical image data represents a medical image focusing on imaging the skeletal structure of the target vertebra, and the second modal medical image data represents a medical image focusing on imaging the soft tissue structure of the target vertebra;

[0008] Image segmentation processing is performed on the first modality medical image data and the second modality medical image data respectively to obtain a first segmentation dataset and a second segmentation dataset; wherein, the first segmentation dataset includes segmented image data corresponding to multiple vertebral segments of the first modality medical image data; the second segmentation dataset includes segmented image data corresponding to multiple vertebral segments of the second modality medical image data;

[0009] Based on the first segmentation dataset and the second segmentation dataset, a pose transformation information set corresponding to the plurality of vertebral segments is determined; wherein, the pose transformation information set is a collection of information used to characterize the relative pose relationship between the plurality of vertebral segments in the first modality medical image data and the second modality medical image data;

[0010] Based on the pose transformation information set, the first segmented dataset or the second segmented dataset is transformed to obtain the transformed first segmented dataset or the transformed second segmented dataset.

[0011] The transformed first segmented dataset is subjected to image stitching processing to obtain a first target image, or the transformed second segmented dataset is subjected to image stitching processing to obtain a second target image; wherein, the first target image is used to represent the image obtained after registering the first modality medical image data with the second modality medical image data as a reference, and the second target image is used to represent the image obtained after registering the second modality medical image data with the first modality medical image data as a reference.

[0012] In a second aspect, the present invention provides an electronic device, comprising: a memory, and one or more processors communicatively connected to the memory; the memory stores instructions executable by the one or more processors, the instructions being executed by the one or more processors to cause the one or more processors to implement the method described above.

[0013] Thirdly, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0014] Beneficial effects:

[0015] This invention significantly reduces the computational complexity and technical overhead of direct full-segment registration by decomposing the medical image registration problem of the entire spine into segmentation, pose transformation, and stitching of multiple vertebral segments. Specifically, it leverages the respective advantages of first and second modalities of medical images in skeletal and soft tissue imaging to achieve accurate vertebral segmentation. By determining the pose transformation information set based on the segmentation results, the calculation of the transformation matrix is ​​further optimized, ensuring the efficiency and accuracy of the registration process. Finally, by weighted fusion and stitching of the transformed segmented dataset, a complete registered image is generated. This overcomes the excessive computational burden caused by overall processing in related technologies, making this method highly real-time and widely applicable, particularly suitable for intraoperative real-time registration applications. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating an image data processing method provided in an embodiment of the present invention;

[0017] Figure 2 This is a flowchart illustrating an image data processing method provided in an embodiment of the present invention;

[0018] Figure 3This is a flowchart illustrating an image data processing method provided in an embodiment of the present invention;

[0019] Figure 4 This is a scene example diagram of an image data processing method provided in an embodiment of the present invention;

[0020] Figure 5 This is a block diagram of an electronic device used in an embodiment of the present invention. Detailed Implementation

[0021] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0022] With the rapid development of medical imaging technology, image fusion and registration are increasingly widely used in modern medicine, especially in scenarios involving the fusion of multimodal medical images (such as CT and MR images), which has significant clinical implications. The goal of multimodal image registration is to align the anatomical structures (such as vertebrae) of the same patient in different modal images, thereby providing physicians with comprehensive anatomical and functional information. This approach can more accurately assist in diagnosis and surgical planning, improving treatment outcomes. However, due to the differences in imaging mechanisms and characteristics between different modal images—CT images have extremely high resolution on skeletal structures, while MR images focus on clear imaging of soft tissue structures—this difference presents a significant challenge to multimodal registration.

[0023] In clinical practice, multimodal image registration of the entire spine is particularly crucial. Because the spine is composed of multiple vertebral segments and has a complex anatomical structure, the registration process requires processing a large amount of 3D image data. Furthermore, the spine often exhibits pose differences in CT and MR images, which may be caused by patient positioning, imaging equipment parameters, or image deformation under different modalities. These differences necessitate extensive global optimization calculations in traditional registration algorithms to achieve accurate vertebral alignment. However, since full-segment registration of the spine typically involves large-scale data processing of dozens of vertebral segments, the registration process significantly increases computational complexity and time overhead, posing a significant technical challenge to clinical applications.

[0024] Related technologies fail to fully utilize the local feature information of vertebral segments in multimodal images, instead relying on time-consuming and labor-intensive global optimization strategies. This approach is inefficient when dealing with large-scale data across the entire spinal segment and cannot meet the requirements for fast and high-precision registration.

[0025] Therefore, in the application scenario of medical image registration of the entire spinal segment, there is a technical problem of excessive computational overhead in related technologies.

[0026] like Figure 1 As shown, an image data processing method is provided, wherein the execution subject of the method may be a processing device, and the method may include:

[0027] Step S12: Receive first modal medical image data and second modal medical image data representing the target vertebral body; wherein, the first modal medical image represents a medical image focusing on imaging the skeletal structure of the target vertebral body, and the second modal medical image represents a medical image focusing on imaging the soft tissue structure of the target vertebral body.

[0028] In this embodiment, the first modality of medical image data may be CT (Computed Tomography) image data.

[0029] In this embodiment, the first modality medical image data can also be other X-ray image data.

[0030] In this embodiment, the first modality of medical image data can also be image data acquired by the C-arm during surgery.

[0031] In this embodiment, the second modality medical image data may be MR (Magnetic Resonance) image data.

[0032] In some implementations, the second modality medical image data may also be ultrasound images or PET (positron emission tomography) image data.

[0033] In this embodiment, the receiving action can be represented as the processing device receiving first modality medical image data and second modality medical image data transmitted from an external device.

[0034] Specifically, the external device can be a database, storage system, network, or image acquisition device. This image acquisition device is used to acquire first-modality medical image data and second-modality medical image data; for example, a CT scanner, MR imaging equipment, etc.

[0035] In this embodiment, the first modality of medical image data and the second modality of medical image data can be stored in DICOM format (Digital Imaging and Communications in Medicine).

[0036] Specifically, DICOM is an internationally standardized medical image storage and transmission format. Each DICOM file contains not only medical image data but also detailed metadata information, which may include the image's position, capture time, modality type, and patient information. The DICOM protocol standardizes the spatial positioning of images. The X-axis points from the patient's right to left; the Y-axis points from the patient's front to back; and the Z-axis points from the patient's feet to their head. This unified coordinate system allows for a consistent reference frame for the spatial positioning of multimodal medical images (e.g., CT and MR images), simplifying the initial alignment process for different modalities. Furthermore, leveraging the prior consistency of the DICOM coordinate system, the initial transformation matrix can be quickly calculated by extracting only a single feature point (e.g., the vertebral body center point or a specific anatomical landmark).

[0037] In this embodiment, the first modality medical image data and the second modality medical image data can also be stored in the MetaImage format.

[0038] In this embodiment, the first modal medical image data and the second modal medical image data can also be stored in the NIfTI format.

[0039] In this embodiment, the target vertebral body refers to the vertebral body of the target patient.

[0040] Step S14: Perform image segmentation processing on the first modality medical image data and the second modality medical image data respectively to obtain a first segmentation dataset and a second segmentation dataset; wherein, the first segmentation dataset includes segmented image data corresponding to multiple vertebral segments of the first modality medical image; the second segmentation dataset includes segmented image data corresponding to multiple vertebral segments of the second modality medical image.

[0041] In this embodiment, the first modality medical image data and the second modality medical image data can be respectively input into a preset image segmentation model for processing to obtain first segmentation information and second segmentation information. Then, based on the first segmentation information, the first modality medical image data is segmented to obtain the first segmented dataset. And based on the second segmentation information, the second modality medical image data is segmented to obtain the second segmented dataset.

[0042] In this embodiment, the first segmentation information and the second segmentation information can be probability graphs.

[0043] In this embodiment, the first segmentation information and the second segmentation information can be feature point coordinates. These feature point coordinates can be the coordinates of key anatomical points of the vertebral segment (e.g., the center of the vertebral body, the spinous process, the transverse process, etc.).

[0044] In this embodiment, the first segmentation information and the second segmentation information can be boundary information. For example, they can be the outer contour information of the target vertebral segment, which may include a set of boundary points or a boundary line.

[0045] In this embodiment, the first segmentation information and the second segmentation information can be bounding boxes. For example, they can be the smallest rectangular or cubic region that surrounds the target vertebral segment.

[0046] In this embodiment, the first segmentation information and the second segmentation information can be region features. These region features can be density histograms, average intensity, or texture features, etc.

[0047] In this embodiment, the first segmentation information and the second segmentation information can also be voxel distribution information.

[0048] In this embodiment, the preset image segmentation model can be a pre-trained deep learning model.

[0049] Specifically, the deep learning model can be the U-Net model.

[0050] The deep learning model mentioned can also be the V-Net model.

[0051] The deep learning model mentioned can also be the Mask R-CNN model.

[0052] It is understood that training data (e.g., labeled CT and MR images) can be pre-collected to train an initial model (one of the deep learning models mentioned above) to obtain an image segmentation model corresponding to the first modality of medical image data and an image segmentation model corresponding to the second modality of medical image data.

[0053] It is understood that when using a pre-trained deep learning model to perform image segmentation processing on the first and second modal medical image data, the first and second segmentation information can be a mask or a probability map. The mask can be a binary image, where 1 indicates belonging to the target vertebral segment and 0 indicates background. The probability map can represent the probability value of each pixel or voxel belonging to the target. The first and second segmentation datasets can be segmented images containing multiple vertebral segments, each segment represented by a separate image or label.

[0054] In this embodiment, an image segmentation algorithm can also be used to perform image segmentation processing on the first modality medical image data and the second modality medical image data respectively to obtain a first segmentation dataset and a second segmentation dataset.

[0055] Specifically, the image segmentation algorithm can be an edge detection-based image segmentation algorithm.

[0056] More specifically, firstly, edge detection algorithms can be used to extract the vertebral body edges from the first and second modal medical image data. Then, the watershed algorithm is used to further segment the vertebral body segments.

[0057] The edge detection algorithm can be based on the Canny edge detection algorithm.

[0058] The edge detection algorithm can be based on the Sobel edge detection algorithm.

[0059] The edge detection algorithm can be a Laplacian-based edge detection algorithm.

[0060] The image segmentation algorithm can also be a region growing method, for example, starting from high-density voxels in the skeletal region and gradually expanding to complete the segmentation of the vertebral body.

[0061] The image segmentation algorithm described can also be a threshold-based segmentation algorithm. For example, a density threshold (such as CT value) can be set to distinguish bone from other tissues.

[0062] It is understandable that when using an edge detection-based image segmentation algorithm to perform image segmentation processing on the first modality of medical image data and the second modality of medical image data, the first segmentation information and the second segmentation information can be the results of edge detection, such as the extracted boundary line image.

[0063] When using region growing to perform image segmentation on the first modality of medical image data and the second modality of medical image data, the first segmentation information can be the initial region of the skeleton (seed point). The second segmentation information can be the highlighted soft tissue region in MR.

[0064] In this embodiment, a hybrid method can also be used to perform image segmentation processing on the first modality medical image data and the second modality medical image data respectively to obtain a first segmentation dataset and a second segmentation dataset.

[0065] Specifically, an initial mask can be provided first using edge detection or region growing. Then, U-Net or V-Net can be used to optimize the segmentation results.

[0066] In this embodiment, a multimodal fusion model can also be used to perform image segmentation processing on the first modality medical image data and the second modality medical image data respectively to obtain a first segmentation dataset and a second segmentation dataset.

[0067] In this embodiment, the segmented image data in the first and second segmentation datasets can be images obtained purely by segmenting using the first and second segmentation information. For example, the first and second segmentation information (segmentation mask, probability map, boundary information) can be used directly as a basis for segmenting the first and second modal image data. This results in a segmented image that directly contains multiple vertebral segments, each segment being independently represented, which can be used for subsequent pose transformation or registration.

[0068] In this embodiment, the segmented image data in the first and second segmentation datasets can be images obtained by expanding the segmentation based on the boundary information provided by the first and second segmentation information. Specifically, the segmented region can be expanded based on the boundaries provided by the first and second segmentation information. Morphological operations can be used, such as dilation, to expand the segmented region, for example, by adding a width of 3mm outside the boundary. Adaptive expansion is also possible; for example, the expansion range can be dynamically adjusted according to the uncertainty of the segmented region or the surrounding structure.

[0069] Step S16: Based on the first segmentation dataset and the second segmentation dataset, determine the pose transformation information set corresponding to the plurality of vertebral segments; wherein, the pose transformation information set is a collection of information used to characterize the relative pose relationship between the plurality of vertebral segments in the first modality medical image data and the second modality medical image data.

[0070] In this embodiment, the pose transformation information set can be a collection of transformation matrices. For example, a rigid transformation matrix can be used to characterize the rotational and translational relationships between vertebral segments. This transformation matrix can include rotation matrices and translation vectors. The rotation matrix can describe the rotational relationship between two modes. The translation vector can describe the positional offset between two modes.

[0071] In this embodiment, the pose transformation information set corresponding to the multiple vertebral segments can be determined in the following way: First, vertebral segment pairing can be performed. Each vertebral segment corresponding to the second segmentation dataset is found in the first segmentation dataset. For example, the vertebral numbers (e.g., L1, L2, L3) labeled in the first and second segmentation datasets can be used for direct pairing. Then, feature point or feature region extraction is performed. That is, key geometric information of the vertebral body in each modality is extracted for calculating pose relationships. The center point (centroid) of the vertebral body can be extracted to calculate the geometric center of the segmentation mask. Anatomical landmarks (e.g., spinous processes, transverse processes) can also be extracted. Finally, pose relationship calculation is performed. The pose transformation relationship between the first and second modalities can be calculated based on the extracted feature points or feature regions.

[0072] More specifically, pose transformation relationships can be calculated based on feature points. Alternatively, they can be calculated based on feature regions. The calculated pose transformation relationships are stored in an information set, which can be indexed by segment number or anatomical location to obtain the pose transformation information set.

[0073] In this embodiment, the pose transformation information set can be a set of pose parameters. These pose parameters can be Euler angles, quaternions, etc.

[0074] In this embodiment, the pose transformation information set can be a set of matching points. The matching point set can describe the geometric relationship between two modal images using corresponding keypoints or feature point pairs.

[0075] In this embodiment, the pose transformation information set can support two registration reference methods: using the first modality medical image data as a reference to transform the second modality medical image data, or using the second modality medical image data as a reference to transform the first modality medical image data.

[0076] Specifically, when it is necessary to use the skeletal structure from CT scans as the primary reference (e.g., during intraoperative navigation), the second modality of medical image data can be transformed based on the first modality of medical image data.

[0077] Specifically, when it is necessary to use MR soft tissue information as the primary reference (e.g., during preoperative planning), the first modality of medical image data can be transformed based on the second modality of medical image data.

[0078] Step S18: Based on the pose transformation information set, perform transformation processing on the first segmentation dataset or the second segmentation dataset to obtain the transformed first segmentation dataset or the transformed second segmentation dataset.

[0079] In this embodiment, the first segmented dataset can be transformed based on the pose transformation information set to obtain the transformed first segmented dataset.

[0080] In this embodiment, the second segmented dataset can also be transformed based on the pose transformation information set to obtain the transformed second segmented dataset.

[0081] Step S110: Perform image stitching processing on the transformed first segmented dataset to obtain first target image data, or perform image stitching processing on the transformed second segmented dataset to obtain second target image data; wherein, the first target image data is used to represent image data obtained by registering the first modality medical image data with the second modality medical image data as a reference, and the second target image data is used to represent image data obtained by registering the second modality medical image data with the first modality medical image data as a reference.

[0082] In this embodiment, the transformed first segmented dataset can be processed by image stitching to obtain the first target image data.

[0083] In this embodiment, the transformed second segmented dataset can also be processed by image stitching to obtain the second target image data.

[0084] In one possible and specific implementation, image stitching can be performed as follows: First, ensure that all cone segments in the transformed segmentation dataset are aligned to the coordinate system of the target segmentation dataset. Then, construct blanking regions in the target image. Initialize the target image's data structure to have the same resolution and size as the baseline modality. Blanking regions can be filled with default values ​​(e.g., 0 or background intensity values). A grid coordinate system for the target image can be constructed based on the baseline modality's resolution and pixel pitch.

[0085] Next, the transformed segmented dataset is traversed, and the data of each vertebral segment is mapped to the coordinate system of the target image.

[0086] Finally, overlapping regions are addressed. In segmented data across multiple vertebral segments, overlapping regions may appear (e.g., the boundary between two segments). Pixel values ​​from these overlapping regions can be fused to avoid stitching artifacts.

[0087] Furthermore, edge smoothing and hole filling can be applied to the stitched images to enhance visual continuity.

[0088] This implementation significantly reduces the computational complexity and technical overhead of direct full-segment registration by decomposing the medical image registration problem of the entire spine into segmentation, pose transformation, and stitching of multiple vertebral segments. Specifically, it leverages the advantages of the first and second modalities of medical images in skeletal and soft tissue imaging to achieve accurate vertebral segmentation. By determining the pose transformation information set based on the segmentation results, the calculation of the transformation matrix is ​​further optimized, ensuring the efficiency and accuracy of the registration process. Finally, by weighted fusion and stitching of the transformed segmented dataset, a complete registered image is generated. This overcomes the computational burden caused by overall processing in related technologies, making this method highly real-time and widely applicable, particularly suitable for intraoperative real-time registration applications.

[0089] like Figure 2 As shown, in some embodiments, the step of performing image segmentation processing on the first modality medical image data and the second modality medical image data respectively to obtain a first segmentation dataset and a second segmentation dataset includes:

[0090] Step S142: Input the first modal medical image data and the second modal medical image data into a preset image segmentation model for processing to obtain first segmentation information and second segmentation information; wherein, the first segmentation information represents the image feature information of the vertebral segment in the first modal medical image data; the second segmentation information represents the image feature information of the vertebral segment in the second modal medical image data.

[0091] In this embodiment, the image segmentation model can be the U-Net model.

[0092] In this embodiment, the image segmentation model can be the V-Net model.

[0093] In this embodiment, the image segmentation model can be the Mask R-CNN model.

[0094] In this embodiment, the image segmentation model can also be a pre-trained multimodal large model. For example, it can be a segmentation model based on a combination of a large language model (LLM) and a visual model.

[0095] In this embodiment, the image segmentation model can also be a segmentation model based on traditional machine learning. For example, it can be an SVM segmentation model.

[0096] In this embodiment, the image segmentation model can also be a hybrid model combining segmentation algorithms and deep learning.

[0097] In this embodiment, the first segmentation information and the second segmentation information may be mask information, probability map, feature point information, boundary information, bounding box information, region features, topological structure information, voxel distribution information, anatomical annotation information, and segmentation confidence, etc.

[0098] Step S144: Based on the first segmentation information, the first modality medical image data is segmented to obtain the first segmentation dataset.

[0099] Step S146: Based on the second segmentation information, the second modality medical image data is segmented to obtain the second segmentation dataset.

[0100] This embodiment processes first and second modal medical image data by inputting them into a preset image segmentation model, respectively, to extract image feature information (first segmentation information and second segmentation information) of vertebral segments. Based on the extracted feature information, the images are segmented to generate a first segmentation dataset and a second segmentation dataset. This method automates the extraction and segmentation of feature information from different modalities of medical images using a preset segmentation model, effectively improving segmentation accuracy and efficiency. Simultaneously, by extracting feature information for different modalities, it fully utilizes the clear imaging of skeletal structures in the first modality and the sensitivity of soft tissue structures in the second modality, providing high-quality input data for subsequent vertebral registration and image stitching, reducing errors caused by human intervention, and enhancing robustness.

[0101] Furthermore, the first segmentation information is represented as the first image mask information of the vertebral segment in the first modality of medical image data, and the second segmentation information is represented as the second image mask information of the vertebral segment in the second modality of medical image data;

[0102] like Figure 3 As shown, the step of segmenting the first modality medical image data based on the first segmentation information to obtain the first segmented dataset includes:

[0103] Step S1442: Perform dilation processing on the first image mask information to obtain the dilated first image mask information.

[0104] In this embodiment, the first image mask information may be image feature information representing the vertebral segment in the first modality of medical image data (e.g., CT image or X-ray image). It may take the form of a binary image that identifies the spatial location and extent of the vertebral segment.

[0105] In this embodiment, a preset dilation kernel can be used to process the first image mask information. This dilation kernel can be rectangular, circular, or elliptical. The size of the dilation kernel can be preset based on the dimensions of the vertebral segment.

[0106] Understandably, dilation serves two purposes: firstly, it can fill in boundary discontinuities caused by noise during segmentation; secondly, it ensures that the segmented target region completely covers the vertebral segment, especially in cases of blurred boundaries or low image quality.

[0107] Step S1444: Use the dilated first image mask information to segment the first modal medical image data to obtain segmented image data of vertebral segments in the first modal medical image data;

[0108] The step of segmenting the second modality medical image data based on the second segmentation information to obtain the second segmentation dataset includes:

[0109] Step S1462: Perform dilation processing on the second image mask information to obtain the dilated second image mask information.

[0110] Step S1464: Use the dilated second image mask information to segment the first modality medical image data to obtain segmented image data of vertebral segments in the second modality medical image data.

[0111] This embodiment dilates the first and second image mask information to generate dilated image mask information. The dilated mask is then used to segment the first and second modal medical image data, respectively. The advantage of this method is that dilation effectively compensates for missing target regions caused by segmentation errors, noise, or incomplete boundaries in the original mask information, thereby improving the integrity and robustness of vertebral segment segmentation. By expanding the segmentation range, sufficient boundary information is ensured in the segmented vertebral segment images across different modalities, providing more accurate and comprehensive input data for subsequent registration and stitching processes, thus improving the overall effect of multimodal fusion processing.

[0112] In some implementations, the pose transformation information set is a set representing transformation matrices; wherein, the transformation matrix represents a matrix capable of transforming the poses of multiple vertebral segments in the first modality of medical image data to the poses of corresponding vertebral segments in the second modality of medical image data;

[0113] The step of determining the pose transformation information set corresponding to the plurality of vertebral segments based on the first segmentation dataset and the second segmentation dataset includes:

[0114] Step S162: Traverse the first segmentation dataset and the second segmentation dataset respectively to extract at least one feature point data corresponding to each segmented image data in the first segmentation dataset and the second segmentation dataset, to obtain a first feature point dataset corresponding to the first segmentation dataset and a second feature point dataset corresponding to the second segmentation dataset.

[0115] In this embodiment, the feature point data can be the geometric center of the vertebral body, or anatomical landmarks of the vertebral body. For example, it can be the tip of the spinous process or the end of the transverse process. The feature point data can also be a set of boundary points of the vertebral body.

[0116] In this embodiment, the geometric center of the cone can be obtained by calculating the geometric center of all pixels or voxels within the segmentation mask area.

[0117] In this embodiment, anatomical landmarks and boundary point sets can be located using edge detection algorithms (e.g., Canny or Sobel) or deep learning models.

[0118] Step S164: Generate a set of transformation matrices based on the second feature point dataset of the first feature point dataset and the second segmentation dataset.

[0119] In one possible and specific implementation, a segmented image data in the first segmentation dataset and a segmented image data in the second segmentation dataset are used (it is understood that these two segmented image data should correspond and represent the same vertebral segment).

[0120] The set of transformation matrices can be generated using the following method.

[0121] First, iterate through the first segmentation dataset and the second segmentation dataset respectively to obtain the first feature point dataset P. first Second feature point dataset Q second

[0122] Then, for each pair of vertebral segment feature points p and q, the translation vector is calculated.

[0123] vec = pq

[0124] In the formula, vec represents the offset by which the feature point p of the first mode is transformed into the feature point q of the second mode.

[0125] It should be noted that when multiple feature points are extracted from a vertebral segment, these feature points are used to calculate the initial rotation matrix and translation vector by minimizing the following loss function:

[0126]

[0127] In the formula, R * Let t represent the optimal solution of the rotation matrix. * p represents the optimal solution for the translation vector. i Let q represent the coordinates of i feature points in a segmented image from the first segmentation dataset. i Let N represent the coordinates of the i-th feature point in a segmented image data from the second segmentation dataset. N represents the total number of feature points in the segmented image data.

[0128] Finally, the transformation matrix is ​​generated.

[0129]

[0130] In the formula, T represents the generated transformation matrix, and R and t represent the rotation matrix and translation vector obtained through optimization, respectively.

[0131] It is understandable that the set of transformation matrices can be generated by traversing the second feature point dataset of the first feature point dataset and the second segmentation dataset.

[0132] In some implementations, the step of traversing the first segmentation dataset and the second segmentation dataset respectively to extract at least one feature point data corresponding to each segmented image data in the first segmentation dataset and the second segmentation dataset, to obtain a first feature point dataset corresponding to the first segmentation dataset and a second feature point dataset corresponding to the second segmentation dataset, includes:

[0133] Step S1622: Determine whether the position of the target vertebra in the first modal medical image data and the first modal medical image data are the same.

[0134] In this embodiment, the position of the target vertebra in the first modal medical image data and the corresponding metadata can be identified to determine whether they are the same.

[0135] Understandably, metadata can carry the target vertebral body's positional information. Specifically, the associated DICOM metadata can be extracted from the first and second modality medical image data. Key fields are then extracted from the metadata. These key fields are then parsed to obtain the target vertebral body's positional description, such as "supine," "prone," "left lateral decubitus," or "right lateral decubitus," etc. The positional information fields in the two sets of metadata are then compared to determine if they are consistent. If the field information is consistent, the target vertebral body's position is considered the same; if the field information is inconsistent, the target vertebral body's position is considered different.

[0136] Step S1624: Under the same conditions, traverse the first segmentation dataset and the second segmentation dataset respectively to extract a feature point data corresponding to each segmented image data in the first segmentation dataset and the second segmentation dataset, so as to obtain the first feature point dataset corresponding to the first segmentation dataset and the second feature point dataset corresponding to the second segmentation dataset.

[0137] Step S1626: In case of different conditions, traverse the first segmentation dataset and the second segmentation dataset respectively to extract multiple feature point data corresponding to each segmented image data in the first segmentation dataset and the second segmentation dataset, to obtain the first feature point dataset corresponding to the first segmentation dataset and the second feature point dataset corresponding to the second segmentation dataset.

[0138] This implementation introduces a mechanism to determine whether the target vertebral body positions are the same, flexibly adjusting the feature point extraction strategy to adapt to different positional relationships. Specifically, when the positions are the same, extracting a single feature point simplifies the calculation process, reduces computational complexity, and improves efficiency. Conversely, when the positions are different, extracting multiple feature points increases the distribution density and diversity of feature points, enhancing the robustness and accuracy of transformation matrix calculations during registration, thereby improving the overall registration effect. This targeted approach effectively balances the trade-off between efficiency and accuracy, adapting to diverse real-world scenarios while improving the algorithm's versatility and reliability.

[0139] In this embodiment, the step of transforming the first segmented dataset or the second segmented dataset based on the pose transformation information set to obtain the transformed first segmented dataset or the transformed second segmented dataset includes:

[0140] The set of transformation matrices is traversed to perform transformations on the segmented image data in the first segmentation dataset, resulting in the transformed first segmentation dataset.

[0141] Determine whether the image matching degree between the transformed first segmented dataset and the second segmented dataset meets a preset similarity threshold.

[0142] In this implementation, the mean square error between the corresponding pixel values ​​of the transformed first segmented dataset and the second segmented dataset can be calculated to determine whether the image matching degree between the transformed first segmented dataset and the second segmented dataset meets a preset similarity threshold. If the error is less than the preset threshold, the matching degree is considered to meet the requirements.

[0143] In this implementation, the structural similarity index (SSIM) between the transformed first segmented dataset and the second segmented dataset can also be calculated to determine whether the image matching degree between the transformed first segmented dataset and the second segmented dataset meets a preset similarity threshold. It is understood that SSIM can be used to measure the similarity of two images in terms of brightness, contrast, and structure. If the SSIM value is greater than the preset threshold, the two images are considered to have a high degree of matching.

[0144] In this embodiment, histogram similarity can also be used to measure the degree of image matching.

[0145] In this embodiment, feature point matching can also be used to measure the degree of image matching.

[0146] In this embodiment, the correlation coefficient can also be used to measure the degree of image matching.

[0147] If the similarity threshold is not met, the transformation matrix is ​​optimized using a preset optimization algorithm so that the image matching degree between the transformed first segmented dataset and the second segmented dataset meets the preset similarity threshold, thereby obtaining the final set of transformation matrices.

[0148] In this embodiment, the optimization algorithm may be the gradient descent algorithm.

[0149] In this embodiment, the optimization algorithm may be the stochastic gradient descent algorithm.

[0150] In this embodiment, the optimization algorithm can be the simulated annealing algorithm.

[0151] In this embodiment, the optimization algorithm can also be a genetic algorithm.

[0152] In this embodiment, the optimization algorithm can also be a particle swarm optimization algorithm.

[0153] The set of final transformation matrices is traversed to perform transformations on the segmented image data in the first segmentation dataset, resulting in the transformed first segmentation dataset.

[0154] In this embodiment, the first segmented dataset is initially transformed by traversing the set of transformation matrices. Then, based on the degree of image matching with the second segmented dataset, a preset similarity threshold is determined. A preset optimization algorithm is used to optimize transformation matrices that do not meet the similarity threshold, thereby obtaining the final set of transformation matrices and completing the final image transformation process. This method dynamically adjusts the transformation matrices, gradually optimizing the matching effect between the first and second segmented datasets, significantly improving the accuracy and reliability of registration. Simultaneously, the automated optimization process reduces the cost of manual intervention and repeated trials, improving processing efficiency, and is particularly suitable for complex multimodal medical image registration tasks involving spinal segments.

[0155] In some implementations, the step of determining whether the image matching degree between the transformed first segmentation dataset and the second segmentation dataset meets a preset similarity threshold includes:

[0156] Based on the maximum mutual information method, it is determined whether the image matching degree between the transformed first segmentation dataset and the second segmentation dataset meets the preset similarity threshold.

[0157] This implementation method uses the maximum mutual information method to determine whether the image matching degree between the transformed first and second segmentation datasets meets a preset similarity threshold. It fully leverages the robustness and adaptability of the maximum mutual information method in multimodal medical image registration, effectively solving the technical challenge of traditional methods' difficulty in assessing the matching degree when there are significant differences in grayscale distribution between images of different modalities. This method can comprehensively evaluate the global and local consistency of two segmentation datasets without relying on features of a single modality image, improving the accuracy and reliability of the registration results. Simultaneously, it reduces the error risk caused by noise and modal differences during multimodal image registration, thereby ensuring the overall accuracy and quality of image registration.

[0158] In one possible and specific implementation, the above method may include:

[0159] Step 1: Input the reference image and the image to be registered. The reference image can be an MRI image. The image to be registered can be an X-ray or CT modal image that needs to be aligned with the reference image.

[0160] Step 2: Set up the registration framework. You can define the type of transformation required to align the image to be registered with the reference image. For example, it could be a rigid body transformation. You can define the interpolation method. The interpolation method can resample the image to be registered during the transformation process; specifically, it can be nearest neighbor interpolation, bilinear interpolation, or cubic interpolation. You can set a similarity measure. For example, it could be mutual information or mean squared error.

[0161] Step 3: Roughly align the two images using the initial transformation matrix obtained in the above steps. This initial transformation generates a preliminary transformation result for the images to be registered.

[0162] Step 4: Calculate mutual information using the following formula.

[0163] I(A,B)=H(A)+H(B)-H(A,B)

[0164] In the formula, H(A) and H(B) are the entropies of image A and image B, respectively, and H(A,B) is the joint entropy of the two images.

[0165] H(A)=-∫p A (a)logp A (a)da

[0166] In the formula, p A (a) represents the probability density function of pixel value a in image A, logp A (a) Used to calculate the uncertainty of pixel value distribution. H(A) is used to measure the randomness or information content of pixel distribution in image A. The higher the randomness, the greater the entropy value.

[0167] H(A,B)=-∫∫p AB (a,b)logp AB (a,b)dadb

[0168] In the formula, p AB (a,b) is the joint probability density function, representing the probability that pixel values ​​a and b occur simultaneously. H(A,B) measures the uncertainty of the joint pixel distribution of two images. If the two images are completely independent, the joint entropy is equal to the sum of the entropies of the two images.

[0169] Step 5: Perform optimization transformation. Adjust the transformation parameters (rotation and translation) using an optimization algorithm. Maximize the similarity metric (e.g., mutual information) to achieve the best registration result.

[0170] Step 6: Output the final transformation result.

[0171] In some implementations, the step of optimizing the transformation matrix using a preset optimization algorithm when the similarity threshold is not met includes:

[0172] If the similarity threshold is not met, the transformation matrix is ​​optimized using a preset conjugate gradient descent algorithm.

[0173] This implementation method optimizes the transformation matrix using a pre-defined conjugate gradient descent algorithm when the similarity threshold is not met. This fully leverages the efficiency and stability of the conjugate gradient algorithm in high-dimensional optimization problems, significantly improving the convergence speed of the optimization process while effectively avoiding the oscillations and slow convergence issues that may occur with simple gradient descent. This method reduces computational overhead while ensuring that the optimized transformation matrix accurately adjusts the matching degree between segmented datasets, thereby meeting the pre-defined similarity threshold. This further enhances the accuracy and robustness of multimodal medical image registration, providing more reliable basic data support for subsequent medical diagnosis and processing.

[0174] In some implementations, the step of performing image stitching processing on the transformed first segmented dataset to obtain the first target image includes:

[0175] The first segmentation dataset is traversed to determine the overlapping regions of adjacent segmented image data.

[0176] Based on the overlapping region, a fusion weight is calculated; wherein, the fusion weight is used to represent the proportion of a pixel in the overlapping region to belong to two adjacent segmented image data, and the closer a pixel is to a certain segmented image data, the more the fusion weight is inclined to that segmented image data.

[0177] In this embodiment, the fusion weight can be used to smooth the weight values ​​of overlapping regions of adjacent segmented image data, which represents the degree of a pixel in the overlapping region relative to two adjacent segmented image data.

[0178] The fusion weight can be in the range [0, 1]. The closer the weight value of a pixel is to 1, the higher the proportion of that pixel belonging to one segment of image data; the closer the weight value is to 0, the lower the proportion of that pixel belonging to another segment of image data.

[0179] By weighted fusion of pixels in overlapping regions, a smooth transition can be achieved, avoiding obvious boundaries or breaks at the stitching points. The fusion weights can be calculated based on the distance of a pixel to the boundary of an adjacent segmented image; the closer the distance, the higher the weight of that segmented image.

[0180] The first target image is obtained by weighting and fusing the pixels of two adjacent segmented image data in the overlapping region based on the fusion weight.

[0181] This implementation performs image stitching on the transformed first segmented dataset and employs a fusion weight calculation method for overlapping regions to refine the attribution ratio of pixels in the overlapping regions to adjacent segmented image data. By assigning higher weights to pixels that are closer to the segmented image data, the smoothness and consistency of the transition region are ensured, effectively avoiding the obvious seam problem caused by direct stitching. This method can improve the image stitching quality while maintaining the integrity of the overall image structure, generating a more continuous and natural first target image. It provides technical support for high-precision fusion of multimodal medical images, thereby improving the reliability of image analysis and diagnosis.

[0182] like Figure 4 As shown, in one possible and specific implementation, the above method may include:

[0183] Step 1: For two adjacent segmented image data, for example, we can define them as the first image and the second image. Determine their overlapping region and obtain the coordinates of each pixel in the overlapping region. We can compare the spatial positions of the two images to determine the location and boundaries of the overlapping region in the adjacent segmented image data. As shown in the figure, the overlapping region is the part where the first image and the second image intersect.

[0184] Step 2: Calculate the fusion weights.

[0185] For each pixel in the overlapping region, calculate its distance to the boundary between the first and second images. For any overlapping pixel p0, define its distance to the nearest boundary of the first image as d1. x d1 y d1 z The distance to the nearest boundary of the second image is d2. x d2 y d2 z .

[0186]

[0187] In the formula, the fusion weight *w* indicates whether a pixel is more likely to belong to the first or second image. The closer a pixel is to a particular image, the more its weight is biased towards that image. By calculating the weights of pixels to the boundaries of adjacent segmented images, a weighted average of pixel values ​​in the overlapping region is achieved, resulting in a smooth transition and avoiding visual abruptness and information loss.

[0188] Step 3: Perform weighted fusion.

[0189] The following formula is used to perform a weighted calculation on each pixel in the overlapping region.

[0190] p0 = p1·w + p2·(1-w)

[0191] In the formula, the fused pixel value p0 combines the pixel information of the first and second images in the overlapping area, resulting in a smooth transition and avoiding obvious stitching marks.

[0192] It is understandable that during multi-segment image stitching, especially between registration results of different segments, overlapping areas may exist. Direct stitching can lead to obvious boundaries or breaks in the image at these overlapping areas. This method, through weighted fusion processing based on weights, ensures a continuous transition of pixel values ​​in the overlapping areas, achieving smooth image stitching. Moreover, in medical diagnosis, complete and flawless spinal images are crucial. This method effectively avoids image discontinuities at the stitching points, thereby generating complete, high-quality spinal images and providing doctors with a more reliable image basis for diagnosis.

[0193] This implementation extracts feature point data from each segmented image data by traversing the first and second segmentation datasets, generating a first feature point dataset and a second feature point dataset respectively. Then, a set of transformation matrices is generated based on these two datasets to describe the pose relationship of vertebral segments between the two modal medical image data. This implementation efficiently establishes a precise registration relationship between the first and second modalities through feature point extraction and matching, reducing the computational overhead of direct full-image registration. Using feature points to generate transformation matrices improves the flexibility and robustness of registration, making it particularly suitable for complex multimodal scenarios. Furthermore, this method has strong tolerance for boundary errors in the segmented regions, ensuring that the generated set of transformation matrices accurately reflects the relative poses of multiple vertebral segments, providing a reliable foundation for subsequent image fusion and diagnosis.

[0194] The registration method of this invention decomposes the overall registration of the spine into single-segment registration. By utilizing the rigidity characteristics of the vertebral skeleton, only rigidity transformation needs to be considered, cleverly avoiding the deformation field set that may occur between vertebrae.

[0195] According to an embodiment of the present invention, an electronic device is provided; please refer to... Figure 5 The electronic device in this embodiment may include one or more of the following components: a processor, a network interface, memory, non-volatile memory, and one or more application programs, wherein the one or more application programs may be stored in non-volatile memory and configured to be executed by one or more processors, and the one or more programs are configured to perform the methods as described in the foregoing method embodiments.

[0196] According to embodiments of the present invention, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a computer, causes the computer to perform the method described in any of the above embodiments.

[0197] According to embodiments of the present invention, a computer program product comprising instructions is also provided, which, when executed by a computer, cause the computer to perform a method in any of the above embodiments.

[0198] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0199] Optionally, specific examples in this embodiment can refer to the examples described in the above embodiments, and will not be repeated here. The sequence numbers of the embodiments in this application are merely for description and do not represent the superiority or inferiority of the embodiments. In the above embodiments of this application, the descriptions of each embodiment have their own emphasis, and parts not described in detail in a certain embodiment can be referred to the relevant descriptions in other embodiments.

[0200] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. An image data processing method, characterized in that, The method includes: Receive first modal medical image data and second modal medical image data representing a target vertebra; the first modal medical image data represents a medical image focusing on imaging the skeletal structure of the target vertebra, and the second modal medical image data represents a medical image focusing on imaging the soft tissue structure of the target vertebra; Image segmentation processing is performed on the first modality medical image data and the second modality medical image data respectively to obtain a first segmentation dataset and a second segmentation dataset; wherein, the first segmentation dataset includes segmented image data corresponding to multiple vertebral segments of the first modality medical image data; the second segmentation dataset includes segmented image data corresponding to multiple vertebral segments of the second modality medical image data; Based on the first segmentation dataset and the second segmentation dataset, a pose transformation information set corresponding to the plurality of vertebral segments is determined; wherein, the pose transformation information set is a collection of information used to characterize the relative pose relationship between the plurality of vertebral segments in the first modality medical image data and the second modality medical image data; Based on the pose transformation information set, the first segmented dataset or the second segmented dataset is transformed to obtain the transformed first segmented dataset or the transformed second segmented dataset. The transformed first segmented dataset is subjected to image stitching processing to obtain a first target image, or the transformed second segmented dataset is subjected to image stitching processing to obtain a second target image; wherein, the first target image is used to represent the image obtained after registering the first modality medical image data with the second modality medical image data as a reference, and the second target image is used to represent the image obtained after registering the second modality medical image data with the first modality medical image data as a reference.

2. The method according to claim 1, characterized in that, The step of performing image segmentation processing on the first modality medical image data and the second modality medical image data respectively to obtain a first segmentation dataset and a second segmentation dataset includes: The first modality of medical image data and the second modality of medical image data are respectively input into a preset image segmentation model for processing to obtain first segmentation information and second segmentation information; wherein, the first segmentation information represents the image feature information of the vertebral segment in the first modality of medical image data; and the second segmentation information represents the image feature information of the vertebral segment in the second modality of medical image data. Based on the first segmentation information, the first modality medical image data is segmented to obtain the first segmentation dataset; Based on the second segmentation information, the second modality medical image data is segmented to obtain the second segmentation dataset.

3. The method according to claim 2, characterized in that, The first segmentation information represents the first image mask information of the vertebral segment in the first modality of medical image data, and the second segmentation information represents the second image mask information of the vertebral segment in the second modality of medical image data; The step of segmenting the first modality medical image data based on the first segmentation information to obtain the first segmented dataset includes: The first image mask information is subjected to dilation processing to obtain the dilated first image mask information; The first modality medical image data is segmented using the dilated first image mask information to obtain segmented image data of vertebral segments in the first modality medical image data. The step of segmenting the second modality medical image data based on the second segmentation information to obtain the second segmentation dataset includes: The second image mask information is subjected to dilation processing to obtain the dilated second image mask information; The second modality medical image data is segmented using the dilated second image mask information to obtain segmented image data of vertebral segments in the second modality medical image data.

4. The method according to claim 1, characterized in that, The pose transformation information set is a set representing transformation matrices; wherein, the transformation matrix is ​​a matrix that can transform the pose of multiple vertebral segments in the first modality of medical image data to the pose of the corresponding vertebral segments in the second modality of medical image data; The step of determining the pose transformation information set corresponding to the plurality of vertebral segments based on the first segmentation dataset and the second segmentation dataset includes: The first segmentation dataset and the second segmentation dataset are traversed respectively to extract at least one feature point data corresponding to each segmented image data in the first segmentation dataset and the second segmentation dataset, so as to obtain a first feature point dataset corresponding to the first segmentation dataset and a second feature point dataset corresponding to the second segmentation dataset. The set of transformation matrices is generated based on the second feature point dataset of the first feature point dataset and the second segmentation dataset.

5. The method according to claim 4, characterized in that, The step of traversing the first segmentation dataset and the second segmentation dataset respectively to extract at least one feature point data corresponding to each segmented image data in the first segmentation dataset and the second segmentation dataset, to obtain a first feature point dataset corresponding to the first segmentation dataset and a second feature point dataset corresponding to the second segmentation dataset, includes: Determine whether the position of the target vertebra in the first modal medical image data and the first modal medical image data are the same; Under the same conditions, the first segmentation dataset and the second segmentation dataset are traversed respectively to extract a feature point data corresponding to each segmented image data in the first segmentation dataset and the second segmentation dataset, so as to obtain the first feature point dataset corresponding to the first segmentation dataset and the second feature point dataset corresponding to the second segmentation dataset. In cases where they are different, the first segmentation dataset and the second segmentation dataset are traversed separately to extract multiple feature point data corresponding to each segmented image data in the first segmentation dataset and the second segmentation dataset, thereby obtaining a first feature point dataset corresponding to the first segmentation dataset and a second feature point dataset corresponding to the second segmentation dataset.

6. The method according to claim 5, characterized in that, The step of transforming the first segmented dataset or the second segmented dataset based on the pose transformation information set to obtain the transformed first segmented dataset or the transformed second segmented dataset includes: The set of transformation matrices is traversed to perform transformations on the segmented image data in the first segmentation dataset, resulting in the transformed first segmentation dataset. Determine whether the image matching degree between the transformed first segmented dataset and the second segmented dataset meets a preset similarity threshold; If the similarity threshold is not met, the transformation matrix is ​​optimized using a preset optimization algorithm so that the image matching degree between the transformed first segmented dataset and the second segmented dataset meets the preset similarity threshold, thereby obtaining the final set of transformation matrices. The set of final transformation matrices is traversed to perform transformations on the segmented image data in the first segmentation dataset, resulting in the transformed first segmentation dataset.

7. The method according to claim 6, characterized in that, The step of determining whether the image matching degree between the transformed first segmented dataset and the second segmented dataset meets a preset similarity threshold includes: Based on the maximum mutual information method, it is determined whether the image matching degree between the transformed first segmentation dataset and the second segmentation dataset meets the preset similarity threshold.

8. The method according to claim 6, characterized in that, In the case where the similarity threshold is not met, the step of optimizing the transformation matrix using a preset optimization algorithm includes: If the similarity threshold is not met, the transformation matrix is ​​optimized using a preset conjugate gradient descent algorithm.

9. The method according to claim 1, characterized in that, The step of performing image stitching processing on the transformed first segmented dataset to obtain the first target image includes: Traverse the first segmentation dataset to determine the overlapping regions of adjacent segmented image data; Based on the overlapping region, a fusion weight is calculated; wherein, the fusion weight is used to represent the proportion of a pixel in the overlapping region to the two adjacent segmented image data; the closer a pixel is to a certain segmented image data, the more the fusion weight is inclined towards that segmented image data. The first target image is obtained by weighting and fusing the pixels of two adjacent segmented image data in the overlapping region based on the fusion weight.

10. An electronic device, characterized in that, include: A memory, and one or more processors communicatively connected to the memory; The memory stores instructions that can be executed by the one or more processors to cause the one or more processors to implement the method as described in any one of claims 1 to 9.

11. A computer-readable storage medium, characterized in that, The readable storage medium stores a computer program that, when executed by a processor, implements the method of any one of claims 1 to 9.

Images