Multimodal image processing method, apparatus, and electronic device

Abstract

The application provides a multi-modal image processing method and device and electronic equipment, which can be applied to the technical field of medical image processing. The method comprises the following steps: based on a coronal section image and a sagittal section image representing a current state of a target object, performing pose conversion on an intermediate conversion section image representing a state of the target object at a historical moment to obtain a target registration section image representing the current state of the target object, wherein the intermediate conversion section image is determined by the following manner: based on a first magnetic resonance section image, performing three-dimensional pixel point mapping on a second magnetic resonance section image to obtain an initial conversion section image mapped to a space coordinate of the first magnetic resonance section image; and based on a three-dimensional contrast section image, performing three-dimensional pixel point mapping on the initial conversion section image to obtain the intermediate conversion section image mapped to a space coordinate of the three-dimensional contrast section image.

Description

Technical Field

[0001] This invention relates to the field of medical image processing technology, and more specifically to a multimodal image processing method, apparatus, and electronic device. Background Technology

[0002] Currently, endovascular interventional surgery is required to treat patients with severe intracranial arterial stenosis and significant symptoms of cerebral ischemia. However, due to the tortuous spatial course and complex lesion morphology of intracranial arteries, surgeons cannot directly observe the three-dimensional course of the vessels from a two-dimensional perspective. This can lead to damage to the vessel wall during guidewire delivery, increasing the difficulty and risk of the surgery. Summary of the Invention

[0003] In view of the above problems, the present invention provides a multimodal image processing method, apparatus and electronic device.

[0004] According to a first aspect of the present invention, a multimodal image processing method is provided, comprising: performing pose transformation on an intermediate transformed segmentation image representing the historical state of the target object based on a coronal segmentation image and a sagittal segmentation image representing the current state of the target object, to obtain a target registration segmentation image representing the current state of the target object, wherein the coronal segmentation image and the sagittal segmentation image are obtained by performing two-dimensional angiography of the target object in different orientations using a rotating imaging device; wherein the intermediate transformed segmentation image is determined by: performing three-dimensional pixel mapping on a second magnetic resonance segmentation image based on a first magnetic resonance segmentation image to obtain an initial transformed segmentation image mapped to the spatial coordinates of the first magnetic resonance segmentation image, wherein the first magnetic resonance segmentation image and the second magnetic resonance segmentation image are obtained by performing three-dimensional information acquisition on the target object using different pulse mechanisms of the same magnetic resonance imaging device; and performing three-dimensional pixel mapping on the initial transformed segmentation image based on the three-dimensional angiography segmentation image to obtain an intermediate transformed segmentation image mapped to the spatial coordinates of the three-dimensional angiography segmentation image, wherein the three-dimensional angiography segmentation image is obtained by performing three-dimensional angiography on the target object using a rotating imaging device.

[0005] A second aspect of the present invention provides a multimodal image processing apparatus, comprising: a pose conversion module, configured to perform pose conversion on an intermediate converted segmentation image representing the historical state of the target object based on a coronal segmentation image and a sagittal segmentation image representing the current state of the target object, to obtain a target registration segmentation image representing the current state of the target object, wherein the coronal segmentation image and the sagittal segmentation image are obtained by performing two-dimensional angiography of the target object in different orientations using a rotating imaging device; wherein the intermediate converted segmentation image is determined by: mapping three-dimensional pixels of a second magnetic resonance segmentation image based on a first magnetic resonance segmentation image to obtain an initial converted segmentation image mapped to the spatial coordinates of the first magnetic resonance segmentation image, wherein the first magnetic resonance segmentation image and the second magnetic resonance segmentation image are obtained by performing three-dimensional information acquisition on the target object using different pulse mechanisms of the same magnetic resonance imaging device; and mapping three-dimensional pixels of the initial converted segmentation image based on a three-dimensional angiography segmentation image to obtain an intermediate converted segmentation image mapped to the spatial coordinates of the three-dimensional angiography segmentation image, wherein the three-dimensional angiography segmentation image is obtained by performing three-dimensional angiography of the target object using a rotating imaging device.

[0006] A third aspect of the present invention provides an electronic device comprising: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors perform the method described above.

[0007] According to the multimodal image processing method, apparatus, and electronic device provided by the present invention, by performing three-dimensional pixel mapping on a second magnetic resonance imaging (MRI) segmentation image based on a first MRI segmentation image, the wall information of missing or stenotic segments in the first MRI segmentation image is supplemented under the same spatial coordinates, thus obtaining complete vascular spatial course information. Furthermore, by performing three-dimensional pixel mapping on the initial transformed segmentation image based on a three-dimensional angiography segmentation image, the luminal morphology is corrected under the same spatial coordinates. Therefore, by fusion of the pose registration of the three modal images—the three-dimensional angiography segmentation image, the first MRI segmentation image, and the second MRI segmentation image—a complete vascular space characterizing the historical state of intracranial vessels of the target object is obtained. The intermediate segmented images representing the course and luminal morphology of the intracranial vessels are transformed. Then, based on the real-time acquired coronal and sagittal segmented images representing the current state of the intracranial vessels, the pose of the intermediate segmented images representing the historical state is calibrated. This corrects the actual pose difference between the 3D images representing the historical state of the intracranial vessels and the 2D images representing the current state of the intracranial vessels. It can determine the pose of the intermediate segmented images in the current state, and thus obtain the transformed 3D image representing the current state of the intracranial vessels (target registration segmented image). This enables real-time precise 3D image navigation, providing surgeons with multi-faceted, multi-dimensional, and multi-view information on the course of stenotic vessels, reducing the operational difficulty and risk factor of intracranial arterial interventional surgery. Attached Figure Description

[0008] The above and other objects, features and advantages of the present invention will become clearer from the following description of embodiments of the invention with reference to the accompanying drawings.

[0009] Figure 1 A flowchart of a multimodal image processing method according to an embodiment of the present invention is shown.

[0010] Figure 2 A schematic diagram illustrating the generation of a target spatial pose transformation vector according to an embodiment of the present invention is shown.

[0011] Figure 3 A schematic diagram illustrating the generation of an initial spatial pose transformation vector and an intermediate spatial pose transformation vector according to an embodiment of the present invention is shown.

[0012] Figure 4 A structural block diagram of a multimodal image processing apparatus according to an embodiment of the present invention is shown.

[0013] Figure 5 A block diagram of an electronic device suitable for implementing a multimodal image processing method according to an embodiment of the present invention is shown. Detailed Implementation

[0014] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the invention. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the invention for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concept of the invention.

[0015] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. The terms “comprising,” “including,” etc., as used herein indicate the presence of features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0016] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0017] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0018] In the process of realizing this invention, it was discovered that due to the tortuous spatial course of intracranial arteries and the complex lesion morphology, the surgeon cannot intuitively observe the three-dimensional course of the blood vessels from a two-dimensional perspective. Moreover, the three-dimensional images are acquired in a historical state, such as before the operation. Since the pose of the target object changes during the operation, the pose of the three-dimensional image in the historical state differs from the actual pose in the current state. This makes it impossible to achieve accurate three-dimensional navigation in the current state during the operation, which leads to damage to the blood vessel wall by the guidewire tip during delivery, increasing the difficulty and risk of the operation.

[0019] In view of this, embodiments of the present invention provide a multimodal image processing method, apparatus, and electronic device. The method includes: performing pose transformation on an intermediate transformed segmentation image representing the historical state of the target object based on a coronal segmentation image and a sagittal segmentation image representing the current state of the target object, to obtain a target registration segmentation image representing the current state of the target object. The coronal and sagittal segmentation images are obtained by performing two-dimensional angiography of the target object from different orientations using a rotating imaging device. The intermediate transformed segmentation image is determined as follows: based on a first magnetic resonance segmentation image, a second magnetic resonance segmentation image is mapped to three-dimensional pixels to obtain an initial transformed segmentation image mapped to the spatial coordinates of the first magnetic resonance segmentation image. The first and second magnetic resonance segmentation images are obtained by acquiring three-dimensional information about the target object using different pulse mechanisms of the same magnetic resonance imaging device. Based on the three-dimensional angiography segmentation image, the initial transformed segmentation image is mapped to three-dimensional pixels to obtain an intermediate transformed segmentation image mapped to the spatial coordinates of the three-dimensional angiography segmentation image. The three-dimensional angiography segmentation image is obtained by performing three-dimensional angiography of the target object using a rotating imaging device.

[0020] In some cases of extreme stenosis or occlusion, the lumen of the diseased vessel segment may not be visualized in the first MRI segmentation image, but the vessel wall may be visualized in the second MRI segmentation image. Therefore, this invention simultaneously maps and fuses pixels from both three-dimensional modal images to determine the overall course of the vessel and the morphological structure of the lesion. The three-dimensional angiography segmentation image reflects blood flow and lumen morphology that closely approximates the current state of the target object in the two-dimensional image. Therefore, using the three-dimensional angiography segmentation image allows for further observation of the lumen morphology at the stenotic site, achieving correction of the lumen morphology within the same spatial coordinate system.

[0021] Because the spatial course of intracranial arteries is tortuous and the lesion morphology is complex, it is impossible to intuitively observe the three-dimensional course of blood vessels from a two-dimensional perspective. This may cause damage to the blood vessel wall during guidewire delivery, or even perforation of the blood vessel, leading to serious surgical complications and increasing the difficulty and risk of surgery. Therefore, this invention determines the target spatial pose transformation vector required for spatial coordinate system registration of the intermediate transformed segmented image after characterizing the historical state with the coronal and sagittal segmented images obtained from the current state. Then, based on the target spatial pose transformation vector, the target registered segmented image is obtained, correcting the actual pose difference between the three-dimensional image in the historical state and the two-dimensional image in the current state, and realizing real-time accurate navigation of the three-dimensional image in the current state.

[0022] From the perspective of the overall technical means of this invention, multimodal images from multiple different types of imaging devices are fused into a single spatial coordinate system, which facilitates the analysis of target object information and improves the aggregation capability of image information. In the technical solution of this invention, the user information (including but not limited to user personal information, user image information, user device information, such as location information) and data (including but not limited to data used for analysis, stored data, and displayed data) involved are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of related data all comply with relevant laws, regulations, and standards, necessary confidentiality measures have been taken, public order and good morals are not violated, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0023] It should be noted that the sequence numbers of the operations in the following methods are for descriptive purposes only and should not be considered as indicating the execution order of the operations. Unless explicitly stated otherwise, the method does not need to be executed in the exact order shown.

[0024] Figure 1 A flowchart of a multimodal image processing method according to an embodiment of the present invention is shown.

[0025] like Figure 1 As shown, the multimodal image processing method 100 includes operation S110.

[0026] In operation S110, based on the coronal and sagittal segmentation images representing the current state of the target object, pose transformation is performed on the intermediate transformation segmentation image representing the historical state of the target object to obtain the target registration segmentation image representing the current state of the target object. The intermediate transformation segmentation image is determined as follows: based on the first magnetic resonance segmentation image, three-dimensional pixel mapping is performed on the second magnetic resonance segmentation image to obtain the initial transformation segmentation image mapped to the spatial coordinates of the first magnetic resonance segmentation image. The first and second magnetic resonance segmentation images are obtained by acquiring three-dimensional information of the target object using different pulse mechanisms of the same magnetic resonance imaging device.

[0027] The first and second MRI segmentation images were obtained by acquiring three-dimensional information of the target object and segmenting diseased vascular tissue using different pulse mechanisms of the same MRI equipment. Both the first and second MRI segmentation images were obtained by contrast imaging using the same MRI equipment and are different images from the same contrast imaging session.

[0028] The target group can be patients waiting to undergo intracranial artery interventional surgery.

[0029] The current state represents the state during the surgical operation on the target object, while the historical state represents the state before the surgical operation on the target object.

[0030] The first segmented magnetic resonance image characterizes the luminal structure and blood flow of intracranial vessels acquired at historical moments, and can be used to determine the overall course of the vessels and the location of stenosis or occlusion. For example, the first segmented magnetic resonance image is a time-of-flight magnetic resonance angiography (TOF-MRA).

[0031] The second magnetic resonance segmentation image represents the intracranial vessel wall information obtained from the state at a historical moment. For example, the second magnetic resonance segmentation image is a high-resolution magnetic resonance imaging (HR-MRI).

[0032] The process of obtaining an initial transformed segmented image by mapping three-dimensional pixels of a second magnetic resonance segmented image based on a first magnetic resonance segmented image may include: setting the first magnetic resonance segmented image as a fixed image, setting the second magnetic resonance segmented image as a floating image to be registered, registering the second magnetic resonance segmented image to the pose in the spatial coordinates of the first magnetic resonance segmented image, thereby supplementing the vessel wall information of the missing or stenotic segments in the first magnetic resonance segmented image to obtain complete spatial course information of the blood vessels, and then fusing the first magnetic resonance segmented image and the registered second magnetic resonance segmented image to obtain the initial transformed segmented image.

[0033] The initial transformed segmented image representation is based on the complete information fused from spatial travel information and pipe wall information after correcting the pose deviation between the first and second magnetic resonance segmented images.

[0034] Three-dimensional contrast-enhanced segmentation images are obtained by performing three-dimensional contrast imaging and segmentation of diseased vascular tissue on the target object using a rotating imaging device.

[0035] Rotating imaging equipment can be used for digital subtraction angiography (DSA).

[0036] Three-dimensional angiography segmentation images represent the three-dimensional anatomical structure information and morphological information such as lesions of intracranial vessels obtained at historical moments. For example, three-dimensional angiography segmentation images are three-dimensional digital subtraction angiography (3D-DSA) images.

[0037] The process of mapping 3D pixels of the initial transformed segmented image to obtain an intermediate transformed segmented image based on the 3D contrast imaging segmented image can include: setting the 3D contrast imaging segmented image as a fixed image, setting the initial transformed segmented image as a floating image to be registered, registering the initial transformed segmented image to the pose in the spatial coordinates of the 3D contrast imaging segmented image, thereby correcting the missing lumen morphology information in the 3D contrast imaging segmented image, and then fusing the 3D contrast imaging segmented image and the registered initial transformed segmented image to obtain the intermediate transformed segmented image.

[0038] The intermediate transformed segmentation image is a complete information fused from the lumen morphology, spatial travel, and wall information after correcting the pose deviation between the 3D contrast segmentation image and the initial transformed segmentation image.

[0039] Intermediate transformation segmentation images are generated in the historical time state so that the pose of the intermediate transformation segmentation images can be directly converted from the coronal and sagittal segmentation images in the current state. This reduces the delay caused by the current state to the multimodal image processing in the historical time state and realizes real-time accurate navigation of the current state's 3D image.

[0040] The coronal and sagittal segmentation images were obtained by performing two-dimensional angiography of the target object from different orientations and segmentation of diseased vascular tissue using a rotating imaging device.

[0041] For example, a coronal segmentation image is obtained by performing an orthogonal two-dimensional angiography of the target object using a rotating imaging device.

[0042] For example, sagittal segmentation image segmentation is obtained by performing lateral two-dimensional imaging of the target object using a rotating imaging device in the current state.

[0043] Three-dimensional angiography segmentation images reflect blood flow states similar to those in coronal or sagittal segmentation images during surgery. However, due to factors such as target object movement and surgical instrument disturbances, the pose of the preoperative three-dimensional angiography segmentation image deviates somewhat from the actual pose of the target object during surgery. Therefore, on the one hand, the blood flow situation in the initial transformed segmentation image can be corrected based on the three-dimensional angiography segmentation image; on the other hand, the intermediate transformed segmentation image obtained based on the corrected three-dimensional angiography segmentation image can serve as a bridge for pose registration between the preoperative three-dimensional image and the intraoperative two-dimensional image.

[0044] Using coronal and sagittal segmented images as fixed images, the intermediate transformed segmented image is set as a floating image to be registered. The intermediate transformed segmented image is registered to the pose in the spatial coordinates of the coronal segmented image and to the pose in the spatial coordinates of the sagittal segmented image, thus obtaining a target registration segmented image that represents the current state of the target object.

[0045] The target registration segmentation image is based on the pose deviation between the coronal segmentation image and the intermediate transformed segmentation image, as well as the pose deviation between the sagittal segmentation image and the intermediate transformed segmentation image, and it also incorporates multimodal information from the 3D contrast imaging segmentation image, the first magnetic resonance segmentation image, and the second magnetic resonance segmentation image.

[0046] Optionally, by performing 3D pixel mapping on the second magnetic resonance imaging (MRI) segmentation image based on the first MRI segmentation image, the wall information of the missing or stenotic segments in the first MRI segmentation image is supplemented under the same spatial coordinates, thus obtaining complete spatial course information of the blood vessels. Then, based on the 3D angiography segmentation image, the initial transformed segmentation image is further mapped with 3D pixels, thereby correcting the luminal morphology under the same spatial coordinates. Therefore, the pose registration and fusion of the three modal images—the 3D angiography segmentation image, the first MRI segmentation image, and the second MRI segmentation image—results in a complete representation of the historical state of the intracranial blood vessels of the target object, showcasing the spatial course of the blood vessels and the luminal morphology. The intermediate segmented image is transformed, and then the pose calibration of the intermediate segmented image of the historical state is performed based on the coronal and sagittal segmented images representing the current state of the intracranial vessels of the target object in real time. This corrects the real pose difference between the three-dimensional image representing the historical state of the intracranial vessels and the two-dimensional image representing the current state of the intracranial vessels. It can determine the pose state of the intermediate segmented image in the current state, and then obtain the transformed three-dimensional image representing the current state of the intracranial vessels (target registration segmented image). This realizes real-time three-dimensional image precise navigation, provides the surgeon with multi-faceted, multi-dimensional, and multi-view information on the course of the stenotic vessels, and reduces the operation difficulty and risk factor of intracranial arterial interventional surgery.

[0047] Optionally, based on the coronal and sagittal segmented images representing the current state of the target object, pose transformation is performed on the intermediate transformed segmented image representing the historical state of the target object to obtain a target registration segmented image representing the current state of the target object. This includes: refining the coronal segmented image, sagittal segmented image, and intermediate transformed segmented image respectively based on a thinning algorithm to obtain a first centerline point corresponding to the coronal segmented image, a second centerline point corresponding to the sagittal segmented image, and a third centerline point corresponding to the intermediate transformed segmented image, wherein the centerline point represents a set of discretely distributed points on the vessel centerline; using the first A dual-projection matching optimization function and a first constraint condition process the first centerline point, the second centerline point, and the third centerline point to obtain the target space pose transformation vector. The first dual-projection matching optimization function is used to optimize the total squared distance obtained based on the sum of squared distances between the first centerline point and the transformed third centerline point, and the sum of squared distances between the second centerline point and the transformed third centerline point. The first constraint condition is used to ensure that the difference between the total squared distances of two adjacent iterations is less than a preset threshold. Based on the target space pose transformation vector, the intermediate transformed segmented image is pose transformed to obtain the target registration segmented image.

[0048] Preprocessing of the coronal, sagittal, and intermediate transformation segmented images includes: performing a closing operation (dilation followed by erosion) on each of the coronal, sagittal, and intermediate transformation segmented images to obtain closed coronal, sagittal, and intermediate transformation segmented images, thereby removing holes in the vascular lumen; and performing an opening operation (erosion followed by dilation) on the closed coronal, sagittal, and intermediate transformation segmented images to obtain preprocessed coronal, sagittal, and intermediate transformation segmented images, thereby removing noise points outside the blood vessels.

[0049] The preprocessed coronal segmentation image is thinned using a thinning algorithm to obtain the first centerline point corresponding to the coronal segmentation image.

[0050] The preprocessed sagittal segmentation image is thinned using a thinning algorithm to obtain the second centerline point corresponding to the sagittal segmentation image.

[0051] The preprocessed intermediate transformation segmentation image is thinned using a thinning algorithm to obtain the third centerline point corresponding to the intermediate transformation segmentation image.

[0052] A thinning algorithm is an algorithm that further refines a initially extracted centerline into a series of continuous, smooth, and accurately positioned single-point sequences. For example, thinning algorithms can include iterative morphological thinning algorithms, statistical neighborhood thinning algorithms, etc.

[0053] The centerline points represent the set of discrete points distributed along the centerline of a blood vessel.

[0054] The first and second centerline points are sets of two-dimensional points, while the third centerline point is a set of three-dimensional points. Therefore, in the two-dimensional point registration process, the third centerline point needs to be projected onto a two-dimensional projection plane to obtain two-dimensional points, and then registered with the first and second centerline points respectively to obtain the target space pose transformation vector. The two-dimensional projection plane can be based on the coronal projection plane or the sagittal projection plane.

[0055] The input data for the first dual-projection matching optimization function consists of the first centerline point, the second centerline point, the third centerline point, and the spatial pose transformation vector for each iteration. The spatial pose transformation vector for each iteration is a variable.

[0056] The third centerline point is then projected onto the two-dimensional projection plane and then transformed based on the spatial pose transformation vector of the current iteration to obtain the transformed third centerline point.

[0057] In each iteration, the first dual-projection matching optimization function is used to calculate the sum of squared distances between the first centerline point and the transformed third centerline point, as well as the sum of squared distances between the second centerline point and the transformed third centerline point, to obtain the total sum of squared distances.

[0058] Calculate the difference between the total sum of squared distances in the current iteration and the total sum of squared distances in the previous iteration using the first constraint condition.

[0059] If the difference is less than a preset threshold, the spatial pose transformation vector of the current iteration is used as the target spatial pose transformation vector.

[0060] If the difference is greater than or equal to a preset threshold, adjust the spatial pose transformation vector for the next iteration, recalculate the difference between the sum of squared total distances of two adjacent iterations, and stop iterating until the difference is less than the preset threshold.

[0061] The target space pose transformation vector represents the pose changes required for simultaneous registration of points in the intermediate transformed segmented image relative to the poses of points in the coronal and sagittal segmented images. These pose changes include positional and orientational changes. Orientational changes can be rotational angle information.

[0062] The target space pose transformation vector is used to determine the pose correspondence between the intermediate transformed segmented image of the historical state and the coronal and sagittal segmented images of the current state, which is beneficial to realize three-dimensional dynamic navigation during surgery.

[0063] The intermediate transformed segmented image is transformed by the target space pose transformation vector to obtain the target registration segmented image.

[0064] Optionally, the first dual-projection matching optimization function and the first constraint condition cause the pose of the third centerline point to be transformed simultaneously to the pose of the first centerline point and the pose of the second centerline point. This continuously reduces the sum of squared total distances obtained based on the first centerline point, the second centerline point, and the transformed third centerline point. When the sum of squared total distances satisfies the iteration stopping condition, the target spatial pose transformation vector required for spatial coordinate system registration of the intermediate transformed segmented image relative to the coronal and sagittal segmented images is determined. The target spatial pose transformation vector corrects the true pose difference between the historical 3D image and the current 2D image. Based on the target registered segmented image obtained from the target spatial pose transformation vector, real-time accurate navigation of the current 3D image is achieved.

[0065] Optionally, the first centerline point, the second centerline point, and the third centerline point are processed using the first dual-projection matching optimization function and the first constraint condition to obtain the target space pose transformation vector, including: mapping the third centerline point to two-dimensional pixels based on the coronal projection matrix to obtain multiple first mapping points, wherein the coronal projection matrix is ​​a ray intensity distribution matrix generated based on a rotating imaging device configured as an extrinsic parameter of the coronal rotation angle, and the first mapping points represent the mapping points of the third centerline point on the coronal plane; based on a distance matching algorithm, determining the first matching points corresponding to each of the multiple first mapping points from the first centerline points; based on The sagittal projection matrix maps the third centerline point to two-dimensional pixels, resulting in multiple second mapping points. The sagittal projection matrix represents the ray intensity distribution matrix generated by a rotating imaging device configured as an extrinsic parameter of the sagittal rotation angle, and the second mapping points represent the mapping points of the third centerline point on the sagittal plane. Based on a distance matching algorithm, second matching points corresponding to each of the multiple second mapping points are determined from the second centerline points. The multiple first matching points, multiple second matching points, multiple first mapping points, and multiple second mapping points are processed using a first dual-projection matching optimization function and a first constraint condition to obtain the target spatial pose transformation vector.

[0066] The rotating imaging device includes multiple intrinsic parameters and multiple extrinsic parameters. The extrinsic parameter of the rotation angle of the rotating imaging device is configured as a coronal plane rotation angle extrinsic parameter of 0°. The intrinsic parameters can be focal length, scale factor, pixel size, distortion coefficient, etc. An intrinsic parameter matrix is ​​obtained based on multiple intrinsic parameters. The extrinsic parameters can be rotation angle, translation position, etc. An extrinsic parameter matrix is ​​obtained based on multiple extrinsic parameters. A coronal plane projection matrix representing the ray intensity distribution matrix is ​​generated based on the intrinsic parameter matrix and the extrinsic parameter matrix.

[0067] Projecting the third centerline points onto the coronal projection matrix yields the first mapping points corresponding to each of the multiple points on the third centerline.

[0068] The first mapping point represents the mapping point of the third centerline point on the coronal plane. The first mapping point is a two-dimensional point.

[0069] For each of the multiple first mapping points, the distance matching algorithm is used to calculate the distance between the first mapping point and multiple points among the first centerline points, and the point among the first centerline points with the smallest distance to the first mapping point is taken as the first matching point corresponding to the first mapping point.

[0070] The rotation angle extrinsic parameter of the rotating imaging device is configured as a sagittal plane rotation angle extrinsic parameter of 90°. Based on the intrinsic parameter matrix and the extrinsic parameter matrix, a sagittal plane projection matrix characterizing the ray intensity distribution matrix is ​​generated.

[0071] Projecting the third centerline points onto the sagittal projection matrix yields second mapping points corresponding to multiple points on the third centerline.

[0072] The second mapping point represents the mapping point of the third centerline point on the sagittal plane. The second mapping point is a two-dimensional point.

[0073] For each of the multiple second mapping points, the distance matching algorithm is used to calculate the distance between the second mapping point and multiple points among the second centerline points. The point among the second centerline points with the smallest distance to the second mapping point is taken as the second matching point corresponding to the second mapping point.

[0074] By using the first dual-projection matching optimization function and the first constraint condition to process multiple first matching points, multiple second matching points, multiple first mapping points and multiple second mapping points, the target space pose transformation vector is obtained.

[0075] Optionally, the target space pose transformation vector is obtained by processing multiple first matching points, multiple second matching points, multiple first mapping points, and multiple second mapping points using the first dual-projection matching optimization function and the first constraint condition, including: processing the spatial translation variable of the i-th iteration using a spatial translation function to obtain the i-th translation matrix; processing the spatial rotation variable of the i-th iteration using a spatial rotation function to obtain the i-th rotation matrix; transforming the multiple first mapping points based on the i-th translation matrix and the i-th rotation matrix to obtain the first transformation point corresponding to each of the multiple first mapping points; and transforming the multiple second mapping points based on the i-th translation matrix and the i-th rotation matrix to obtain the second transformation point corresponding to each of the multiple second mapping points. For each of the multiple first mapping points, a first sum of squared distances between the first matching point and the first conversion point is determined; for each of the multiple second mapping points, a second sum of squared distances between the second matching point and the second conversion point is determined; based on the first and second sums of squared distances, the total sum of squared distances for the i-th iteration is obtained; if the difference between the total sum of squared distances for the i-th iteration and the total sum of squared distances for the (i-1)-th iteration is less than a preset threshold, the target spatial pose transformation vector is obtained based on the spatial translation variable and the spatial rotation variable for the i-th iteration, where i is an integer greater than 1.

[0076] The spatial pose transformation vector W is based on the spatial translation variables (x, y, z) and the spatial rotation variables (x, y, z). The obtained W = (x, y, z, ...) Spatial translation variables represent the distances that a pixel needs to be translated in the x, y, and z directions, while spatial rotation variables represent the angles that a pixel needs to be rotated in the x, y, and z directions.

[0077] The spatial pose transformation vector in the i-th iteration It is based on the spatial translation variable of the i-th iteration ( ) and the spatial rotation variable of the i-th iteration ( ) obtained, = ( , ). Transform the spatial pose vector Inputting spatial translation and spatial rotation functions yields the pose transformation matrix for the i-th iteration. Based on this pose transformation matrix, multiple second mapping points are transformed.

[0078] In one embodiment, the i-th translation matrix As shown in formula (1):

[0079] (1).

[0080] in,( ) represents the spatial translation variable of the i-th iteration.

[0081] In one embodiment, the i-th rotation matrix As shown in formula (2):

[0082] (2).

[0083] in,( ) represents the spatial rotation variable in the i-th iteration. Characterizing the sine function, Characterizes the cosine function.

[0084] Based on the i-th translation matrix and the i-th rotation matrix, the pose transformation matrix T for the i-th iteration is obtained. )= .

[0085] Based on the pose transformation matrix of the i-th iteration, multiple first mapping points are transformed to obtain the first transformation point corresponding to each of the multiple first mapping points; based on the pose transformation matrix of the i-th iteration, multiple second mapping points are transformed to obtain the second transformation point corresponding to each of the multiple second mapping points.

[0086] The first transformation point represents the point after the first mapping point has undergone spatial translation and spatial rotation transformations.

[0087] The second transformation point represents the point after the second mapping point has undergone spatial translation and spatial rotation transformations.

[0088] For each of the multiple first mapping points, the first matching point and the first transformation point are used as computation pairs to obtain multiple sets of computation pairs. The sum of squared distances between the multiple sets of computation pairs is then calculated.

[0089] For example, for the first mapping points A, B, and C, there are the first matching point A1 and the first transformation point A2 corresponding to the first mapping point A, the first matching point B1 and the first transformation point B2 corresponding to the first mapping point B, and the first matching point C1 and the first transformation point C2 corresponding to the first mapping point C. Statistical functions are used to calculate the squares of the distances between the first matching point A1 and the first transformation point A2, the first matching point B1 and the first transformation point B2, and the first matching point C1 and the first transformation point C2, thus obtaining the first sum of squared distances.

[0090] The calculation process of the second sum of squared distances between the second matching point and the second transformation point is similar to the calculation process of the first sum of squared distances between the first matching point and the first transformation point, and will not be repeated here.

[0091] Based on the sum of the first and second squared distances, the total squared distance of the i-th iteration is obtained. If the difference between the total squared distance of the i-th iteration and the total squared distance of the (i-1)-th iteration is less than a preset threshold, the spatial translation variable based on the i-th iteration ( ) and the spatial rotation variable of the i-th iteration ( The spatial pose transformation vector determined in the i-th iteration. It is determined to be the target space pose transformation vector.

[0092] If the difference between the total squared distance of the i-th iteration and the total squared distance of the (i-1)-th iteration is greater than or equal to a preset threshold, the spatial pose transformation vector of the (i+1)-th iteration is adjusted based on the spatial translation variable and the spatial rotation variable of the i-th iteration for optimization, until the difference between the total squared distance of two adjacent iterations is less than the preset threshold, and the iteration stops.

[0093] In one embodiment, the first dual-projection matching optimization function is shown in equation (3):

[0094] (3).

[0095] in, Let i be the coordinates of the i-th point among the points on the third center line. The number of midpoints of the third center line. It is a 4-dimensional vector space; It is based on the pose parameter vector of the i-th iteration. The calculated pose transformation matrix; Representing the three-dimensional spatial dimension; These are the coronal projection matrix and the sagittal projection matrix, respectively; Let these be the homogeneous coordinates of the first matching points corresponding to each of the multiple first mapping points from the first centerline point. Let these be the coordinates of the second matching points corresponding to each of the multiple second mapping points within the second centerline point. is the target space pose transformation vector.

[0096] Figure 2 A schematic diagram illustrating the generation of a target spatial pose transformation vector according to an embodiment of the present invention is shown.

[0097] like Figure 2As shown, a two-dimensional thinning algorithm is used to thin the coronal segmentation image to obtain the first centerline point corresponding to the coronal segmentation image; the two-dimensional thinning algorithm is also used to thin the sagittal segmentation image to obtain the second centerline point corresponding to the sagittal segmentation image; and a three-dimensional thinning algorithm is used to thin the intermediate transformed segmentation image to obtain the third centerline point corresponding to the intermediate transformed segmentation image. The first dual-projection matching optimization function and the first constraint condition are then used to process the first, second, and third centerline points to obtain the target space pose transformation vector.

[0098] Optionally, the multimodal image processing method further includes: for any point among the third centerline points, using a distance transformation algorithm to calculate the distance from the point to the blood vessel boundary, obtaining the distance values ​​corresponding to each of the multiple points; obtaining the target blood vessel diameter based on the minimum distance value among the multiple distance values; and obtaining the blood vessel inlet radius based on the distance value corresponding to the point with the smallest distance to the bottom of the intermediate transformed segmented image.

[0099] The distance transformation algorithm can be either a distance function or a skewing algorithm. The distance transformation algorithm is used to calculate the distance value from a point to the boundary of a blood vessel.

[0100] A region is a block of points outside the blood vessel. Points within a region have a distance value of 0, points adjacent to the region have smaller distance values, and points farther from the region have larger distance values. The distance from a point to the blood vessel boundary is used as the radius of the blood vessel at that point on the center line.

[0101] The minimum distance value among the distance values ​​corresponding to multiple points is determined as the minimum blood vessel radius, and the value of multiplying the minimum blood vessel radius by 2 is determined as the target blood vessel diameter.

[0102] The target vessel diameter represents the minimum diameter of the vessel. The point corresponding to the minimum distance value is the location of the vessel with the minimum diameter.

[0103] During the angiography process, the intracranial artery inlet is located at the bottom of the image, so the distance value from the point closest to the bottom plane is determined as the vessel inlet radius.

[0104] Optionally, an intermediate transformation segmentation image with multimodal registration and fusion can be acquired before surgery. During the surgical preparation stage, the vascular inlet radius and target vascular diameter can be accurately extracted based on the intermediate transformation segmentation image, thereby effectively assisting staff in selecting appropriate interventional devices.

[0105] Optionally, the multimodal image processing method further includes: for the i-th point among the third centerline points, determining multiple neighboring points corresponding to the i-th point; processing the multiple neighboring points using a fitting algorithm to obtain a fitting curve; if the fitting error of the fitting curve is less than a preset error threshold, processing the first and second derivatives of the i-th point using a curvature function to obtain the curvature of the i-th point; if the curvature of the i-th point is greater than the historical maximum curvature, updating the historical maximum curvature based on the curvature of the i-th point to obtain the target curvature, wherein the historical maximum curvature represents the maximum value among the curvatures corresponding to the first i-1 points, and i is a positive integer greater than 1.

[0106] For each point in the third centerline, local fitting is performed sequentially to obtain multiple fitted curves, and the maximum curvature is updated based on the fitting error of the multiple fitted curves.

[0107] For point i, determine multiple neighboring points corresponding to point i. The fitting algorithm can be the least squares method, which processes multiple neighboring points to obtain the fitting curve corresponding to point i.

[0108] If the fitting error is less than the preset error threshold φ, the curvature function is used to process the first and second derivatives of the i-th point to obtain the curvature of the i-th point.

[0109] In one embodiment, the curvature at point i As shown in formula (4):

[0110] (4).

[0111] in, For point i The first derivative; For point i The second derivative of .

[0112] The maximum curvature among the curvatures of the first i-1 points is determined as the historical maximum curvature. The curvature of the i-th point is compared with the historical maximum curvature. If the curvature of the i-th point is greater than the historical maximum curvature, the historical maximum curvature is updated based on the curvature of the i-th point.

[0113] In related technologies, the curvature of intracranial arteries at various locations needs to be fitted using spline curves. However, the diameter of intracranial arteries often varies at different locations, which can cause problems in the curve fitting process. Spline curves tend to fit areas with larger vessel diameters, resulting in the final fitting result failing to reflect the vessel's course. Furthermore, bifurcation points in the vessels make it impossible to fit them using a single spline curve.

[0114] Optionally, by adopting a local fitting strategy for each point in the third centerline, the obtained neighborhood point set can be kept distributed around a curve along the direction of the vessel centerline. The resulting fitted curve can accurately reflect the vessel shape. Based on the local fitting error, the historical maximum curvature is updated to obtain a precise target curvature. During the operation, the operator can refer to the target curvature when the interventional instrument passes through the vessel segment with a smaller diameter and the vessel segment with a larger curvature to perform the interventional operation, so as to reduce damage to the vessel wall.

[0115] Optionally, based on the first magnetic resonance segmentation image, a three-dimensional pixel mapping is performed on the second magnetic resonance segmentation image to obtain an initial transformed segmentation image mapped to the spatial coordinates of the first magnetic resonance segmentation image. This includes: thinning the first and second magnetic resonance segmentation images respectively based on a thinning algorithm to obtain the centerline points of the first and second magnetic resonance segmentation images; processing the centerline points of the first and second magnetic resonance segmentation images using a first point matching optimization function and a second constraint condition to obtain an initial spatial pose transformation vector, wherein the first point matching optimization function is used to optimize the sum of squared distances between the centerline points of the first magnetic resonance segmentation image and the centerline points of the transformed second magnetic resonance segmentation image, and the second constraint condition is used to constrain the difference between the sum of squared distances of two adjacent iterations to be less than a preset threshold; performing pose transformation on the second magnetic resonance segmentation image based on the initial spatial pose transformation vector to obtain an initial mapped segmentation image mapped to the spatial coordinates of the first magnetic resonance segmentation image; and fusing the initial mapped segmentation image and the first magnetic resonance segmentation image to obtain the initial transformed segmentation image.

[0116] The preprocessing of the first and second magnetic resonance segmented images includes: performing a closing operation on the image (first performing a dilation operation, then performing an erosion operation) to remove holes in the lumen of the blood vessels; and performing an opening operation on the image (first performing an erosion operation, then performing a dilation operation) to remove noise points outside the blood vessels.

[0117] The preprocessed first magnetic resonance segmentation image is thinned using a thinning algorithm to obtain the centerline points corresponding to the first magnetic resonance segmentation image.

[0118] The preprocessed second magnetic resonance segmentation image is thinned using a thinning algorithm to obtain the centerline points corresponding to the second magnetic resonance segmentation image.

[0119] The centerline points of both the first and second magnetic resonance segmented images are sets of three-dimensional points; therefore, three-dimensional point registration can be performed directly without projection.

[0120] The input variables of the first point matching optimization function are the centerline points of the first magnetic resonance segmentation image, the centerline points of the second magnetic resonance segmentation image, and the spatial pose transformation vector for each iteration.

[0121] The centerline point of the transformed second magnetic resonance segmentation image represents the point after the centerline point of the second magnetic resonance segmentation image has undergone pose transformation based on the spatial pose transformation vector of the current iteration.

[0122] In each iteration, the sum of squared distances between the centerline points of the first magnetic resonance segmentation image and the centerline points of the transformed second magnetic resonance segmentation image is calculated using the first point matching optimization function.

[0123] The difference between the sum of squared distances in the current iteration and the sum of squared distances in the previous iteration is calculated using the second constraint. If the difference is less than a preset threshold, the spatial pose transformation vector of the current iteration is used as the initial spatial pose transformation vector.

[0124] If the difference is greater than or equal to a preset threshold, adjust the spatial pose transformation vector for the next iteration, recalculate the difference between the sum of squared distances of two adjacent iterations, and stop iterating until the difference is less than the preset threshold.

[0125] In one embodiment, the first point matching optimization function is shown in formula (5):

[0126] (5).

[0127] in, Here are the homogeneous coordinates of the centerline point of the second magnetic resonance segmentation image. The number of points; It is based on the pose parameter vector of the i-th iteration. The calculated pose transformation matrix, These are the homogeneous coordinates of the points corresponding to the centerline points of the first magnetic resonance segmentation image and the centerline points of the second magnetic resonance segmentation image.

[0128] The initial mapped segmented image represents the second magnetic resonance segmented image, which undergoes pixel pose transformation to be mapped to the segmented image in the spatial coordinates of the first magnetic resonance segmented image.

[0129] The pixel poses of the initial mapped segmentation image and the first magnetic resonance segmentation image have been registered, so the initial mapped segmentation image and the first magnetic resonance segmentation image are fused to obtain the initial transformed segmentation image.

[0130] Optionally, based on the 3D contrast imaging segmentation image, the initial transformed segmentation image is mapped to 3D pixels to obtain an intermediate transformed segmentation image mapped to the spatial coordinates of the 3D contrast imaging segmentation image. This includes: refining the initial transformed segmentation image and the 3D contrast imaging segmentation image respectively based on a thinning algorithm to obtain the centerline points of the initial transformed segmentation image and the 3D contrast imaging segmentation image; processing the centerline points of the initial transformed segmentation image and the 3D contrast imaging segmentation image using a second point matching optimization function and a third constraint condition to obtain an intermediate spatial pose transformation vector, wherein the second point matching optimization function is used to optimize the sum of squared distances between the centerline points of the 3D contrast imaging segmentation image and the centerline points of the transformed initial transformed segmentation image, and the third constraint condition is used to constrain the difference between the sum of squared distances of two adjacent iterations to be less than a preset threshold; performing pose transformation on the initial transformed segmentation image based on the intermediate spatial pose transformation vector to obtain an intermediate mapped segmentation image mapped to the spatial coordinates of the 3D contrast imaging segmentation image; and fusing the intermediate mapped segmentation image and the 3D contrast imaging segmentation image to obtain the intermediate transformed segmentation image.

[0131] The preprocessing of the initial transformed segmented image and the 3D angiography segmented image includes: performing a closing operation on the image (dilation followed by erosion) to remove holes in the blood vessel lumen; and performing an opening operation on the image (erosion followed by dilation) to remove noise points outside the blood vessel.

[0132] The thinning algorithm is used to thin the preprocessed initial transformed segmented image to obtain the centerline points corresponding to the initial transformed segmented image.

[0133] The preprocessed 3D contrast-enhanced image is thinned using a thinning algorithm to obtain the centerline points corresponding to the 3D contrast-enhanced image.

[0134] The centerline points of both the initial transformed segmented image and the 3D imaging segmented image are sets of 3D points. Therefore, 3D point mapping can be performed directly without projection.

[0135] The input variables of the second point matching optimization function are the centerline points of the 3D imaging segmentation image, the centerline points of the initial transformed segmentation image, and the spatial pose transformation vector for each iteration.

[0136] The centerline points of the initial transformed segmented image represent the points after the centerline points of the initial transformed segmented image have undergone pose transformation based on the spatial pose transformation vector of the current iteration.

[0137] In each iteration, the sum of squared distances between the centerline points of the 3D angiographic segmentation image and the centerline points of the transformed initial segmentation image is calculated using the second point matching optimization function.

[0138] The difference between the sum of squared distances in the current iteration and the sum of squared distances in the previous iteration is calculated using the third constraint. If the difference is less than a preset threshold, the spatial pose transformation vector of the current iteration is used as the intermediate spatial pose transformation vector.

[0139] If the difference is greater than or equal to a preset threshold, adjust the spatial pose transformation vector for the next iteration, recalculate the difference between the sum of squared distances of two adjacent iterations, and stop iterating until the difference is less than the preset threshold.

[0140] The intermediate mapping segmentation image represents the initial transformed segmentation image, which undergoes pixel pose transformation to map it to the segmentation image in the spatial coordinates of the 3D imaging segmentation image.

[0141] The pixel poses of the intermediate mapping segmentation image and the 3D contrast segmentation image have been registered, so the intermediate mapping segmentation image and the 3D contrast segmentation image are fused to obtain the intermediate transformed segmentation image.

[0142] Figure 3 A schematic diagram illustrating the generation of an initial spatial pose transformation vector and an intermediate spatial pose transformation vector according to an embodiment of the present invention is shown.

[0143] like Figure 3 As shown, a three-dimensional thinning algorithm is used to thin the first magnetic resonance imaging (MRI) segmentation image, the second MRI segmentation image, and the three-dimensional contrast imaging segmentation image, respectively, to obtain the centerline points of the first MRI segmentation image, the second MRI segmentation image, and the three-dimensional contrast imaging segmentation image. The centerline points of the first and second MRI segmentation images are processed using a first point matching optimization function and a second constraint condition to obtain an initial spatial pose transformation vector. Based on the initial spatial pose transformation vector, the second MRI segmentation image is pose transformed to obtain an initial mapped segmentation image. The initial mapped segmentation image and the first MRI segmentation image are fused to obtain an initial transformed segmentation image. The centerline points of the initial transformed segmentation image and the three-dimensional contrast imaging segmentation image are processed using a second point matching optimization function and a third constraint condition to obtain an intermediate spatial pose transformation vector.

[0144] Based on the above-described multimodal image processing method, this invention also provides a multimodal image processing apparatus. The following will be combined with... Figure 4 The device is described in detail.

[0145] Figure 4 A structural block diagram of a multimodal image processing apparatus according to an embodiment of the present invention is shown.

[0146] like Figure 4 As shown, the multimodal image processing apparatus 400 of this embodiment includes a pose conversion module 410.

[0147] The pose conversion module 410 is used to perform pose conversion on an intermediate converted segmentation image representing the historical state of the target object based on the coronal and sagittal segmentation images representing the current state of the target object, thereby obtaining a target registration segmentation image representing the current state of the target object. The coronal and sagittal segmentation images are obtained by performing two-dimensional angiography of the target object from different orientations using a rotating imaging device. The intermediate converted segmentation image is determined as follows: based on the first magnetic resonance segmentation image, a three-dimensional pixel mapping is performed on the second magnetic resonance segmentation image to obtain an initial converted segmentation image mapped to the spatial coordinates of the first magnetic resonance segmentation image. The first and second magnetic resonance segmentation images are obtained by acquiring three-dimensional information about the target object using different pulse mechanisms of the same magnetic resonance imaging device. Based on the three-dimensional angiography segmentation image, a three-dimensional pixel mapping is performed on the initial converted segmentation image to obtain an intermediate converted segmentation image mapped to the spatial coordinates of the three-dimensional angiography segmentation image. The three-dimensional angiography segmentation image is obtained by performing three-dimensional angiography of the target object using a rotating imaging device.

[0148] Optionally, the pose conversion module 410 includes a first refinement submodule, a first processing submodule, and a first conversion submodule.

[0149] The first thinning submodule is used to thin the coronal segmentation image, sagittal segmentation image and intermediate transformation segmentation image respectively based on the thinning algorithm, to obtain the first centerline point corresponding to the coronal segmentation image, the second centerline point corresponding to the sagittal segmentation image and the third centerline point corresponding to the intermediate transformation segmentation image, wherein the centerline point represents the set of discretely distributed points on the blood vessel centerline.

[0150] The first processing submodule is used to process the first centerline point, the second centerline point, and the third centerline point using the first dual-projection matching optimization function and the first constraint condition to obtain the target space pose transformation vector. The first dual-projection matching optimization function is used to optimize the total squared distance obtained based on the sum of squared distances between the first centerline point and the transformed third centerline point, and the sum of squared distances between the second centerline point and the transformed third centerline point. The first constraint condition is used to constrain the difference between the total squared distances of two adjacent iterations to be less than a preset threshold.

[0151] The first transformation submodule is used to perform pose transformation on the intermediate transformed segmented image based on the target space pose transformation vector to obtain the target registration segmented image.

[0152] Optionally, the first processing submodule includes a first processing unit, a second processing unit, a third processing unit, a fourth processing unit, and a fifth processing unit.

[0153] The first processing unit is used to perform two-dimensional pixel mapping on the third centerline point based on the coronal projection matrix to obtain multiple first mapping points. The coronal projection matrix is ​​a ray intensity distribution matrix generated based on a rotating imaging device configured as an extrinsic parameter of the coronal rotation angle. The first mapping points represent the mapping points of the third centerline point on the coronal plane.

[0154] The second processing unit is used to determine, based on a distance matching algorithm, the first matching point corresponding to each of the multiple first mapping points from the first centerline point.

[0155] The third processing unit is used to perform two-dimensional pixel mapping on the third centerline point based on the sagittal projection matrix to obtain multiple second mapping points. The sagittal projection matrix represents the ray intensity distribution matrix generated by the rotating imaging device based on the sagittal rotation angle extrinsic parameter, and the second mapping points represent the mapping points of the third centerline point on the sagittal plane.

[0156] The fourth processing unit is used to determine the second matching point corresponding to each of the multiple second mapping points from the second centerline point based on the distance matching algorithm.

[0157] The fifth processing unit is used to process multiple first matching points, multiple second matching points, multiple first mapping points and multiple second mapping points using the first dual-projection matching optimization function and the first constraint conditions to obtain the target space pose transformation vector.

[0158] Optionally, the fifth processing unit includes a first processing subunit, a second processing subunit, a third processing subunit, a fourth processing subunit, a fifth processing subunit, a sixth processing subunit, a seventh processing subunit, and an eighth processing subunit.

[0159] The first processing subunit is used to process the spatial translation variable of the i-th iteration using the spatial translation function to obtain the i-th translation matrix.

[0160] The second processing subunit is used to process the spatial rotation variables of the i-th iteration using the spatial rotation function to obtain the i-th rotation matrix.

[0161] The third processing subunit is used to transform multiple first mapping points based on the i-th translation matrix and the i-th rotation matrix to obtain the first transformation point corresponding to each of the multiple first mapping points.

[0162] The fourth processing subunit is used to transform multiple second mapping points based on the i-th translation matrix and the i-th rotation matrix to obtain the second transformation points corresponding to each of the multiple second mapping points.

[0163] The fifth processing subunit is used to determine the first sum of squared distances between the first matching point and the first conversion point for each of the multiple first mapping points.

[0164] The sixth processing subunit is used to determine the second sum of squared distances between the second matching point and the second conversion point for each of the multiple second mapping points.

[0165] The seventh processing subunit is used to obtain the total sum of squared distances for the i-th iteration based on the first sum of squared distances and the second sum of squared distances.

[0166] The eighth processing subunit is used to obtain the target spatial pose transformation vector based on the spatial translation variable and the spatial rotation variable of the i-th iteration when the difference between the total squared distance of the i-th iteration and the total squared distance of the (i-1)-th iteration is less than a preset threshold, where i is an integer greater than 1.

[0167] Optionally, the multimodal image processing apparatus 400 may also include a distance calculation module, an acquisition module, and a selection module.

[0168] The distance calculation module is used to calculate the distance from any point in the third centerline to the blood vessel boundary using a distance transformation algorithm, and obtain the distance values ​​corresponding to each of the multiple points.

[0169] The module is used to obtain the target blood vessel diameter based on the minimum distance value among multiple distance values.

[0170] The selection module is used to obtain the vessel inlet radius based on the distance value corresponding to the point with the smallest bottom distance from the intermediate transformed segmented image.

[0171] Optionally, the multimodal image processing apparatus 400 may also include a determination module, a fitting module, a filtering module, and an update module.

[0172] The determination module is used to determine multiple neighboring points corresponding to the i-th point among the third centerline points.

[0173] The fitting module is used to process multiple neighborhood points using a fitting algorithm to obtain a fitted curve.

[0174] The filtering module is used to process the first and second derivatives of the i-th point using the curvature function when the fitting error of the fitted curve is less than a preset error threshold, so as to obtain the curvature of the i-th point.

[0175] The update module is used to update the historical maximum curvature based on the curvature of point i when the curvature at point i is greater than the historical maximum curvature, so as to obtain the target curvature. Here, the historical maximum curvature represents the maximum value among the curvatures corresponding to the first i-1 points, and i is a positive integer greater than 1.

[0176] Optionally, the pose conversion module 410 may also include a second refinement submodule, a second processing submodule, a second conversion submodule, and a first fusion submodule.

[0177] The second thinning submodule is used to thin the first magnetic resonance segmentation image and the second magnetic resonance segmentation image respectively based on the thinning algorithm, so as to obtain the center line points of the first magnetic resonance segmentation image and the second magnetic resonance segmentation image respectively.

[0178] The second processing submodule is used to process the centerline points of the first magnetic resonance segmentation image and the second magnetic resonance segmentation image using the first point matching optimization function and the second constraint condition to obtain the initial spatial pose transformation vector. The first point matching optimization function is used to optimize the sum of squared distances between the centerline points of the first magnetic resonance segmentation image and the centerline points of the transformed second magnetic resonance segmentation image. The second constraint condition is used to constrain the difference between the sum of squared distances of two adjacent iterations to be less than a preset threshold.

[0179] The second transformation submodule is used to perform pose transformation on the second magnetic resonance segmentation image based on the initial spatial pose transformation vector, so as to obtain an initial mapped segmentation image mapped to the spatial coordinates of the first magnetic resonance segmentation image.

[0180] The first fusion submodule is used to fuse the initial mapped segmentation image and the first magnetic resonance segmentation image to obtain the initial transformed segmentation image.

[0181] Optionally, the pose conversion module 410 may also include a third refinement submodule, a third processing submodule, a third conversion submodule, and a second fusion submodule.

[0182] The third thinning submodule is used to thin the initial transformed segmented image and the 3D contrast-enhanced segmented image respectively based on the thinning algorithm, so as to obtain the centerline points of the initial transformed segmented image and the 3D contrast-enhanced segmented image respectively.

[0183] The third processing submodule is used to process the centerline points of the initial transformed segmented image and the 3D angiography segmented image using the second point matching optimization function and the third constraint condition to obtain the intermediate spatial pose transformation vector. The second point matching optimization function is used to optimize the sum of squared distances between the centerline points of the 3D angiography segmented image and the centerline points of the transformed initial transformed segmented image. The third constraint condition is used to constrain the difference between the sum of squared distances of two adjacent iterations to be less than a preset threshold.

[0184] The third transformation submodule is used to perform pose transformation on the initial transformed segmentation image based on the intermediate spatial pose transformation vector, so as to obtain an intermediate mapped segmentation image in the spatial coordinates of the 3D imaging segmentation image.

[0185] The second fusion submodule is used to fuse the intermediate mapped segmented image and the 3D imaging segmented image to obtain an intermediate transformed segmented image.

[0186] Optionally, any plurality of sub-modules, units, or sub-units in the pose conversion module 410 may be combined into one module, or any one of these modules may be split into multiple modules. Alternatively, at least a portion of the functionality of one or more of these modules may be combined with at least a portion of the functionality of other modules and implemented in one module. Optionally, the pose conversion module 410 may be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or any other reasonable means of integrating or packaging circuitry, or implemented in software, hardware, or firmware, or in any appropriate combination of any of these three implementation methods. Alternatively, the pose conversion module 410 may be partially implemented as a computer program module, which, when run, can perform corresponding functions.

[0187] Figure 5 A block diagram of an electronic device suitable for implementing a multimodal image processing method according to an embodiment of the present invention is shown.

[0188] Figure 5 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of the present invention.

[0189] like Figure 5 As shown, a computer electronic device 500 according to an embodiment of the present invention includes a processor 501, which can perform various appropriate actions and processes according to a program stored in a ROM 502 (read-only memory) or a program loaded from a storage portion 508 into a RAM 503 (random access memory). The processor 501 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 501 may also include onboard memory for caching purposes. The processor 501 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0190] RAM 503 stores various programs and data required for the operation of electronic device 500. Processor 501, ROM 502, and RAM 503 are interconnected via bus 504. Processor 501 executes various operations of the method flow according to embodiments of the present invention by executing programs in ROM 502 and / or RAM 503. It should be noted that programs may also be stored in one or more memories other than ROM 502 and RAM 503. Processor 501 may also execute various operations of the method flow according to embodiments of the present invention by executing programs stored in one or more memories.

[0191] Optionally, the electronic device 500 may also include an input / output (I / O) interface 505, which is also connected to the bus 504. The electronic device 500 may also include one or more of the following components connected to the input / output (I / O) interface 505: an input section 506 including a keyboard, mouse, etc.; an output section 507 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 508 including a hard disk, etc.; and a communication section 509 including a network interface card such as a LAN card, modem, etc. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the input / output (I / O) interface 505 as needed. A removable medium 511, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 510 as needed so that computer programs read from it can be installed into the storage section 508 as needed.

[0192] Optionally, the method flow according to embodiments of the present invention can be implemented as a computer software program. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a computer-readable storage medium, the computer program containing program code for performing the method shown in the flowchart. In such embodiments, the computer program can be downloaded and installed from a network via communication section 509, and / or installed from removable medium 511. When the computer program is executed by processor 501, it performs the functions defined in the system of embodiments of the present invention. Optionally, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0193] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the multimodal image processing method according to embodiments of the present invention.

[0194] Optionally, the computer-readable storage medium can be a non-volatile computer-readable storage medium. Examples include, but are not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this invention, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0195] For example, optionally, the computer-readable storage medium may include the ROM 502 and / or RAM 503 described above and / or one or more memories other than ROM 502 and RAM 503.

[0196] Embodiments of the present invention also include a computer program product comprising a computer program containing program code for performing the methods provided in the embodiments of the present invention. When the computer program product is run on an electronic device, the program code is used to enable the electronic device to implement the multimodal image processing method provided in the embodiments of the present invention.

[0197] When the computer program is executed by the processor 501, it performs the functions defined in the system / apparatus of this embodiment of the invention. Optionally, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0198] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 509, and / or installed from a removable medium 511. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0199] Optionally, program code for executing the computer programs provided in the embodiments of the present invention can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0200] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions. Those skilled in the art will understand that the features described in the various embodiments of the present invention can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in the present invention. In particular, the features described in the various embodiments of the present invention can be combined and / or combined in various ways without departing from the spirit and teachings of the present invention. All such combinations and / or pairings fall within the scope of this invention.

[0201] The embodiments of the present invention have been described above. However, these embodiments are merely illustrative and not intended to limit the scope of the invention. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of the invention, and all such substitutions and modifications should fall within the scope of the invention.

Claims

1. A multimodal image processing method, characterized in that, The method includes: Based on the coronal and sagittal segmented images representing the current state of the target object, pose transformation is performed on the intermediate transformation segmented images representing the historical state of the target object to obtain the target registration segmented image representing the current state of the target object. The coronal and sagittal segmented images are obtained by performing two-dimensional imaging of the target object in different orientations using a rotating imaging device. The intermediate transformation segmentation image is determined in the following way: Based on the first magnetic resonance imaging (MRI) segmentation image, a three-dimensional pixel mapping is performed on the second MRI segmentation image to obtain an initial transformed segmentation image mapped to the spatial coordinates of the first MRI segmentation image. The first and second MRI segmentation images are obtained by acquiring three-dimensional information of the target object using different pulse mechanisms of the same MRI device. The first MRI segmentation image represents the luminal structure and blood flow of intracranial vessels acquired at historical time points, used to determine the overall course of the vessels and the location of stenosis or occlusion. The second MRI segmentation image represents the vessel wall information of intracranial vessels acquired at historical time points. Based on the three-dimensional contrast-enhanced segmentation image, the initial transformed segmentation image is mapped to three-dimensional pixels to obtain an intermediate transformed segmentation image mapped to the spatial coordinates of the three-dimensional contrast-enhanced segmentation image. The three-dimensional contrast-enhanced segmentation image is obtained by performing three-dimensional contrast imaging on the target object using a rotating imaging device.

2. The method according to claim 1, characterized in that, The step of performing pose transformation on the intermediate transformed segmentation image representing the historical state of the target object based on the coronal and sagittal segmentation images representing the current state of the target object, to obtain the target registration segmentation image representing the current state of the target object, includes: Based on the thinning algorithm, the coronal segmentation image, the sagittal segmentation image, and the intermediate transformation segmentation image are thinned respectively to obtain the first centerline point corresponding to the coronal segmentation image, the second centerline point corresponding to the sagittal segmentation image, and the third centerline point corresponding to the intermediate transformation segmentation image, wherein the centerline point represents the set of discretely distributed points on the blood vessel centerline; The first dual-projection matching optimization function and the first constraint condition are used to process the first centerline point, the second centerline point and the third centerline point to obtain the target space pose transformation vector. The first dual-projection matching optimization function is used to optimize the total squared distance obtained based on the squared distance between the first centerline point and the transformed third centerline point and the squared distance between the second centerline point and the transformed third centerline point. The first constraint condition is used to constrain the difference between the total squared distance between two adjacent iterations to be less than a preset threshold. Based on the target space pose transformation vector, the intermediate transformed segmented image is pose transformed to obtain the target registration segmented image; The step of processing the first centerline point, the second centerline point, and the third centerline point using the first dual-projection matching optimization function and the first constraint condition to obtain the target space pose transformation vector includes: Two-dimensional pixel mapping is performed on the third centerline point based on the coronal projection matrix to obtain multiple first mapping points. The coronal projection matrix is ​​a ray intensity distribution matrix generated by a rotating imaging device configured with the external parameter of the coronal rotation angle. The first mapping points represent the mapping points of the third centerline point on the coronal plane. Based on the distance matching algorithm, a first matching point corresponding to each of the multiple first mapping points is determined from the first centerline point; Two-dimensional pixel mapping is performed on the third centerline point based on the sagittal projection matrix to obtain multiple second mapping points. The sagittal projection matrix represents the ray intensity distribution matrix generated by a rotating imaging device configured with sagittal rotation angle extrinsic parameters, and the second mapping points represent the mapping points of the third centerline point on the sagittal plane. Based on the distance matching algorithm, a second matching point corresponding to each of the multiple second mapping points is determined from the second centerline point; The spatial translation variable of the i-th iteration is processed using a spatial translation function to obtain the i-th translation matrix; The spatial rotation variables of the i-th iteration are processed using a spatial rotation function to obtain the i-th rotation matrix; Based on the i-th translation matrix and the i-th rotation matrix, multiple first mapping points are transformed to obtain first transformation points corresponding to each of the multiple first mapping points; Based on the i-th translation matrix and the i-th rotation matrix, multiple second mapping points are transformed to obtain the second transformation points corresponding to each of the multiple second mapping points; For each of the first matching points and first transformation points corresponding to the plurality of first mapping points, determine the first sum of squared distances between the first matching points and the first transformation points; For each of the multiple second mapping points, a second matching point and a second transformation point are corresponding to a second matching point and a second transformation point, and a second sum of squared distances between the second matching point and the second transformation point is determined. Based on the first sum of squared distances and the second sum of squared distances, the total sum of squared distances for the i-th iteration is obtained; If the difference between the total squared distance of the i-th iteration and the total squared distance of the (i-1)-th iteration is less than a preset threshold, the target spatial pose transformation vector is obtained based on the spatial translation variable and the spatial rotation variable of the i-th iteration, where i is an integer greater than 1. The spatial pose transformation vector is obtained based on the spatial translation variable and the spatial rotation variable; the spatial translation variable represents the distance that the pixel needs to translate in the x, y, and z directions, and the spatial rotation variable represents the angle that the pixel needs to rotate in the x, y, and z directions. If the difference between the total squared distance of the i-th iteration and the total squared distance of the (i-1)-th iteration is greater than or equal to a preset threshold, adjust the spatial translation variable and the spatial rotation variable of the i-th iteration to obtain the spatial pose transformation vector of the (i+1)-th iteration, until the difference between the total squared distance of two adjacent iterations is less than the preset threshold, and stop iterating.

3. The method according to claim 2, characterized in that, The method further includes: For any point among the third centerline points, the distance from the point to the blood vessel boundary is calculated using a distance transformation algorithm to obtain the distance values ​​corresponding to each of the multiple points. The target blood vessel diameter is obtained based on the minimum distance value among the multiple distance values. The vessel inlet radius is obtained based on the distance value corresponding to the point with the smallest distance from the bottom of the intermediate transformed segmented image.

4. The method according to claim 2, characterized in that, The method further includes: For the i-th point among the points on the third center line, Determine multiple neighboring points corresponding to the i-th point; A fitting algorithm is used to process multiple neighborhood points to obtain a fitting curve; If the fitting error of the fitted curve is less than a preset error threshold, the first and second derivatives of the i-th point are processed using a curvature function to obtain the curvature of the i-th point. If the curvature at point i is greater than the historical maximum curvature, the historical maximum curvature is updated based on the curvature at point i to obtain the target curvature, wherein the historical maximum curvature represents the maximum value among the curvatures corresponding to the first i-1 points, and i is a positive integer greater than 1.

5. The method according to claim 1, characterized in that, The step of mapping three-dimensional pixels of the second magnetic resonance segmentation image based on the first magnetic resonance segmentation image to obtain an initial transformed segmentation image mapped to the spatial coordinates of the first magnetic resonance segmentation image includes: Based on the thinning algorithm, the first magnetic resonance segmentation image and the second magnetic resonance segmentation image are thinned respectively to obtain the center line points of the first magnetic resonance segmentation image and the second magnetic resonance segmentation image respectively. The centerline points of the first and second magnetic resonance segmented images are processed using a first point matching optimization function and a second constraint condition to obtain an initial spatial pose transformation vector. The input variables of the first point matching optimization function are the centerline points of the first and second magnetic resonance segmented images and the spatial pose transformation vector for each iteration. The transformed centerline point of the second magnetic resonance segmented image represents the point where the centerline point of the second magnetic resonance segmented image undergoes pose transformation based on the spatial pose transformation vector of the current iteration. In each iteration, the first point matching optimization function is used to calculate the sum of squared distances between the centerline points of the first and second magnetic resonance segmented images. The second constraint condition is used to calculate the difference between the sum of squared distances of the current iteration and the sum of squared distances of the previous iteration. If the difference is less than a preset threshold, the spatial pose transformation vector of the current iteration is used as the initial spatial pose transformation vector. If the difference is greater than or equal to the preset threshold, the spatial pose transformation vector for the next iteration is adjusted, and the difference between the sums of squared distances of two adjacent iterations is recalculated until the difference is less than the preset threshold, at which point the iteration stops. Based on the initial spatial pose transformation vector, the second magnetic resonance segmentation image is pose transformed to obtain an initial mapped segmentation image mapped to the spatial coordinates of the first magnetic resonance segmentation image. The initial mapped segmentation image and the first magnetic resonance segmentation image are fused to obtain the initial transformed segmentation image.

6. The method according to claim 1, characterized in that, The process of mapping the initial transformed segmented image to three-dimensional pixels based on the three-dimensional contrast imaging segmentation image to obtain an intermediate transformed segmented image mapped to the spatial coordinates of the three-dimensional contrast imaging segmented image includes: Based on the thinning algorithm, the initial transformed segmented image and the three-dimensional angiography segmented image are thinned respectively to obtain the centerline points of the initial transformed segmented image and the three-dimensional angiography segmented image respectively; The centerline points of the initial transformed segmented image and the three-dimensional angiographic segmented image are processed using the second point matching optimization function and the third constraint condition to obtain the intermediate spatial pose transformation vector. The input variables of the second point matching optimization function are the centerline points of the 3D contrast-enhanced segmentation image, the centerline points of the initial transformed segmentation image, and the spatial pose transformation vector for each iteration. The centerline points of the transformed initial segmentation image represent the points after the centerline points of the initial transformed segmentation image have undergone pose transformation based on the spatial pose transformation vector of the current iteration. In each iteration, the second point matching optimization function is used to calculate the sum of squared distances between the centerline points of the 3D contrast-enhanced segmentation image and the centerline points of the transformed initial segmentation image. The third constraint condition is used to calculate the difference between the sum of squared distances of the current iteration and the sum of squared distances of the previous iteration. If the difference is less than a preset threshold, the spatial pose transformation vector of the current iteration is used as the intermediate spatial pose transformation vector. If the difference is greater than or equal to the preset threshold, the spatial pose transformation vector of the next iteration is adjusted, and the difference between the sum of squared distances of two adjacent iterations is recalculated until the difference is less than the preset threshold, at which point the iteration stops. Based on the intermediate spatial pose transformation vector, the initial transformed segmentation image is transformed to obtain an intermediate mapped segmentation image mapped to the spatial coordinates of the three-dimensional imaging segmentation image. The intermediate mapped segmented image and the three-dimensional imaging segmented image are fused to obtain the intermediate transformed segmented image.

7. A multimodal image processing apparatus, comprising: The pose conversion module is used to perform pose conversion on an intermediate converted segmentation image representing the historical state of the target object, based on the coronal and sagittal segmentation images representing the current state of the target object, to obtain a target registration segmentation image representing the current state of the target object. The coronal and sagittal segmentation images are obtained by performing two-dimensional angiography of the target object from different orientations using a rotating imaging device. The intermediate converted segmentation image is determined as follows: based on a first magnetic resonance segmentation image, a second magnetic resonance segmentation image is mapped to three-dimensional pixels to obtain an initial converted segmentation image mapped to the spatial coordinates of the first magnetic resonance segmentation image. The first and second magnetic resonance segmentation images are obtained by acquiring three-dimensional information of the target object using different pulse mechanisms of the same magnetic resonance imaging device. The first magnetic resonance segmentation image represents the luminal structure and blood flow of intracranial blood vessels acquired at a historical moment, used to determine the overall course of the blood vessels and the location of stenosis or occlusion. The second magnetic resonance segmentation image represents the wall information of intracranial blood vessels acquired at a historical moment. Based on the three-dimensional angiography segmentation image, the initial transformed segmentation image is mapped to three-dimensional pixels to obtain an intermediate transformed segmentation image mapped to the spatial coordinates of the three-dimensional angiography segmentation image. The three-dimensional angiography segmentation image is obtained by performing three-dimensional angiography on the target object using a rotating imaging device.

8. An electronic device, comprising: One or more processors; Memory, used to store one or more computer programs. The characteristic feature is that the one or more processors invoke the one or more computer programs to implement the steps of the method according to any one of claims 1 to 6.

Images