A high-precision intelligent fusion method and system for mpMRI images and ultrasound images

Abstract

The invention belongs to the technical field of medical image analysis, and provides a high-precision intelligent fusion method and system for mpMRI images and ultrasonic images, which obtain mpMRI medical image data and ultrasonic image data; perform initial registration on a floating image and a reference image through affine transformation to obtain an initial position of the floating image; establish a multi-level pyramid decomposition image, start from the highest level of the pyramid, and calculate the deformation field of the highest level of the pyramid; perform similarity measurement to obtain a similarity measurement value; determine whether the similarity measurement value meets a set threshold range to obtain a deformation field at the bottom of the pyramid; apply the deformation field at the bottom of the pyramid to the floating image to obtain a final image registration result, thereby improving the auxiliary value of mpMRI and realizing the automation of fusion and puncture of prostate cancer images and ultrasonic images.

Description

Technical Field

[0001] The present invention belongs to the technical field of medical image analysis, and in particular relates to a high-precision intelligent fusion method and system of mpMRI images and ultrasonic images. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] At present, the high-precision intelligent fusion of prostate cancer mpMRI images and ultrasound images has the following problems:

[0004] 1. The traditional Marching Cubes (MC) algorithm has the disadvantages of easy generation of holes on the model surface, large amount of calculation and low accuracy in the 3D reconstruction of mpMRI medical images of prostate cancer.

[0005] 2. Traditional manual annotation of mpMRI images is too expensive and time-consuming;

[0006] 3. In the traditional Active Demons-based mpMRI and ultrasound image registration method, the grayscale correlation between the two modalities of mpMRI and ultrasound images is poor, and the mutual information alone cannot accurately align the images; the use of ultrasound probes or MRI rectal coils will cause deformation of the prostate, which will lead to differences in the target shapes of the two modality images. Summary of the invention

[0007] In order to solve at least one technical problem existing in the above-mentioned background technology, the present invention provides a high-precision intelligent fusion method and system of mpMRI images and ultrasound images. By associating floating images and reference images, a pyramid decomposition image is constructed, and the correlation between the two modalities of mpMRI and ultrasound images is calculated hierarchically, thereby solving the problem that the correlation between the two modalities of mpMRI and ultrasound images in grayscale is poor, and the image cannot be accurately aligned using mutual information alone.

[0008] The use of an ultrasound probe or an MRI rectal coil can cause deformation of the prostate, which can lead to differences in the target shape between the two modality images.

[0009] In order to achieve the above object, the present invention adopts the following technical solution:

[0010] The first aspect of the present invention provides a high-precision intelligent fusion method of mpMRI images and ultrasound images, comprising the steps of:

[0011] Acquire mpMRI medical imaging data and ultrasound imaging data;

[0012] Using ultrasound image data as reference images, mpMRI medical image data is reconstructed into a three-dimensional model to obtain a floating image;

[0013] Performing initial registration of the floating image and the reference image through affine transformation to obtain the initial position of the floating image;

[0014] Establish a multi-level pyramid decomposition image, starting from the highest level of the pyramid, and calculate the deformation field of the highest level of the pyramid;

[0015] Performing similarity measurement on the initial position of the floating image and the deformation field of the highest layer of the pyramid to obtain a similarity measurement value;

[0016] Determine whether the similarity measurement value meets a set threshold range. If not, change and recalculate the deformation field until the similarity measurement requirement is met. If so, perform interpolation calculation on the deformation field of the highest layer of the pyramid to obtain the deformation field of the bottom of the pyramid.

[0017] The deformation field at the bottom of the pyramid is applied to the floating image to obtain the final image registration result.

[0018] The second aspect of the present invention provides a high-precision intelligent fusion system for mpMRI images and ultrasound images, comprising:

[0019] The data acquisition module is configured to: acquire mpMRI medical image data and ultrasound image data;

[0020] The image fusion module is configured to: use the ultrasound image data as a reference image, reconstruct the mpMRI medical image data into a three-dimensional model based on the MC algorithm to obtain a floating image; perform initial registration of the floating image and the reference image through affine transformation to obtain an initial position of the floating image;

[0021] Establish a multi-level pyramid decomposition image, starting from the highest level of the pyramid, and calculate the deformation field of the highest level of the pyramid;

[0022] Performing similarity measurement on the initial position of the floating image and the deformation field of the highest layer of the pyramid to obtain a similarity measurement value;

[0023] It is determined whether the similarity measurement value meets the set threshold range. If not, the deformation field is changed and recalculated until the similarity measurement requirement is met. If it is met, the deformation field of the highest layer of the pyramid is interpolated to obtain the deformation field of the bottom of the pyramid; the deformation field of the bottom of the pyramid is applied to the floating image to obtain the final image registration result.

[0024] A third aspect of the present invention provides a computer-readable storage medium.

[0025] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps in the high-precision intelligent fusion method of mpMRI images and ultrasound images as described above.

[0026] A fourth aspect of the present invention provides a computer device.

[0027] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps in the high-precision intelligent fusion method of mpMRI images and ultrasound images as described above are implemented.

[0028] Compared with the prior art, the present invention has the following beneficial effects:

[0029] By associating the floating image and the reference image, a pyramid decomposition image is constructed, and the correlation between the two modalities, mpMRI and ultrasound images, is calculated hierarchically. This solves the problem that the grayscale correlation between the two modalities, mpMRI and ultrasound images, is poor and that images cannot be accurately aligned using mutual information alone.

[0030] Determine whether the similarity measurement value meets the set threshold range. If not, change the parameters and recalculate the deformation field u h Until the similarity measurement requirements are met. If they are met, the deformation field of the highest layer of the pyramid is interpolated to obtain the deformation field of the bottom of the pyramid; the deformation field of the bottom of the pyramid is applied to the floating image to obtain the final image registration result, which solves the problem that the use of ultrasound probes or MRI rectal coils will cause deformation of the prostate, resulting in differences in the target shape of the two modal images. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0032] Figure 1 It is a structural framework diagram of the improved high-precision intelligent fusion method of mpMRI and ultrasound images;

[0033] Figure 2 It is the flow chart of the improved Marching Cubes (MC) algorithm;

[0034] Figure 3 This is the framework diagram of the FocalMix method;

[0035] Figure 4 It is an illustrative example of two hybrid methods. DETAILED DESCRIPTION

[0036] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0037] It should be noted that the following detailed descriptions are all illustrative and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0038] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0039] Embodiment 1

[0040] In recent years, the incidence of prostate cancer in China has continued to rise, becoming a major disease threatening the health of elderly men in my country. Therefore, accurate diagnosis of prostate cancer has become particularly important.

[0041] With the development of medical imaging and the continuous improvement of imaging technology, transrectal prostate ultrasound and prostate multiparametric magnetic resonance imaging (mpMRI) examinations have gradually been used in clinical practice.

[0042] Transrectal ultrasound examination has become the most widely used imaging examination method in clinical practice due to its advantages such as real-time, non-invasive and simplicity. However, transrectal ultrasound examination still has certain shortcomings in the detection of prostate cancer lesions. For example, not all lesions of prostate cancer patients have the imaging manifestation of low-echo nodules.

[0043] The accuracy of transrectal ultrasound prostate puncture biopsy will decrease for suspicious lesions of smaller size or located in certain special parts found by mpMRI. Therefore, more accurate, objective and convenient puncture methods are needed in clinical work to guide targeted puncture of the prostate. In addition, the diagnostic sensitivity of multi-parameter MRI in diagnosing prostate cancer can be as high as 90%, and its diagnostic specificity can be as high as 85%. Therefore, mpMRI detection has a very significant advantage in diagnosing prostate cancer in the diagnosis of prostate-related diseases. The innovative fusion-guided puncture method that combines the diagnostic advantages of mpMRI with the real-time and convenience of transrectal ultrasound detection has become an important trend in the development of prostate puncture.

[0044] like Figure 1As shown, this embodiment discloses a high-precision intelligent fusion method of mpMRI images and ultrasound images, including the following steps:

[0045] S1: Acquire mpMRI original image data and real-time ultrasound image data;

[0046] S2: Using the real-time ultrasound image data as the reference image F, the mpMRI medical image data is reconstructed into a three-dimensional model based on the MC algorithm to obtain a floating image M;

[0047] S3: Performing initial registration of the floating image M and the reference image F through affine transformation to obtain an initial position M1 of the floating image;

[0048] S4: Create a multi-level pyramid decomposition image for the floating image M and the reference image F, starting from the highest level h of the pyramid, using M h and F h Represent the h-th layer floating image and reference image respectively, and M h and F h Substituting into formula (1), we can calculate the deformation field u of the highest layer h of the pyramid: h ;

[0049]

[0050] Among them, f and m are the corresponding grayscales in the reference image and the floating image, and are the corresponding gradients in the reference image and the floating image;

[0051] S5: From the initial floating image M1 and the deformation field u h Get a new floating image M h , and then add and into formula (2) (energy function formula) to determine its similarity measurement value;

[0052] E(u(r))=∫ Ω [sim(f(r),m(u(r))+reg(u(r))]dr (2)

[0053] Where u(r) represents the deformation field, m(u(r)) represents the floating image, f(r) represents the reference image, sim(f(r),m(u(r))) represents the similarity measure component, and reg(u(r)) represents the regular component;

[0054] S6: Determine whether the similarity measurement value meets the set threshold range. If not, change the parameters and recalculate the deformation field u h Until the similarity measurement requirements are met. If they are met, the deformation field u of the highest level h of the pyramid is h Interpolate to get the deformation field u of the next layerm , then, F h-1 、M h-1 、u h-1 Put it into the formula and get the deformation field u0 of the bottom of the pyramid;

[0055] S7: Apply the pyramid bottom deformation field u0 to the floating image M to obtain the final image registration result.

[0056] like Figure 2 As shown, the real-time ultrasound image data is used as the reference image F, and the mpMRI medical image data is reconstructed into a three-dimensional model based on the MC algorithm to obtain a floating image M, which includes the following steps:

[0057] S201: scanning two adjacent layers of the mpMRI original image data layer by layer, and constructing cubes one by one;

[0058] S202: Compare the grayscale value of the cube vertex with the threshold of the cube's isosurface to obtain an index value;

[0059] S203: constructing a lookup table by classification, and encapsulating the index table in a protocol mapping table;

[0060] Among them, this embodiment uses 24 topological configurations to be divided into three categories for processing. Each category has its own lookup table, function pointer and processing function to facilitate fast search and processing. Each category is processed in a multi-threaded manner, each category corresponds to a thread, and the synchronization between multiple threads adopts the key segment method;

[0061] The key segment method is specifically: the key segment method is specifically that multiple threads use the same small segment of code, but must obtain ownership of certain shared resources before using it; thereby avoiding the thread from switching from user mode to kernel mode and trying to reduce the consumption of this process.

[0062] For a cube with isosurfaces, the gradient of its vertices is calculated using the difference method, as shown in formula (2):

[0063]

[0064]

[0065]

[0066] Where gx, gy, yz represent the gradients in the x, y, z directions respectively; x, y, z represent the positions of the vertices in the rectangular coordinate system; I(*) represents the function of the vertices of the isosurface cube; Δx, Δy, Δz represent the differences of x, y, z brought into I.

[0067] S204: Using the index value and the multi-threaded parallel processing method, the protocol mapping table points to the corresponding classification edge index table through the function pointer to obtain the intersection edge of the current cube that has an intersection with the isosurface; wherein the multi-threaded synchronization method is implemented using the critical section.

[0068] S205: According to the two vertices and normal vectors of the intersecting edges obtained in the above step S204, the coordinates and normal vectors of the equivalent points are calculated by the midpoint selection method; according to the data in the edge index table, the improved and expanded triangle patch index table is checked through the function pointer in the protocol mapping table to determine the combination of equivalent points that constitute the triangle patch in the current cube.

[0069] The calculation process of the coordinates and normal vector of the equivalent point is as follows: Let A1 and A2 represent the two vertices on a certain edge of the cube, and A represents the midpoint of the edge. Then, the midpoint selection method gives:

[0070]

[0071] Assume its normal vector is n, then by the midpoint selection method we have:

[0072]

[0073] Among them, n1 and n2 represent the normal vectors of the two vertices of the cube voxel respectively.

[0074] S206: Constructing isosurfaces from the triangles in each cube.

[0075] When using artificial intelligence technology to detect lesions in prostate cancer mpMRI medical images, a large number of original nuclear magnetic resonance images need to be annotated. This annotation work not only requires professional expert physicians, but also takes a lot of time and effort. In view of the above problems, this embodiment is based on mpMRI three-dimensional reconstructed medical images, and proposes an image suspicious site marking method based on FocalMix. The features of pathological images are used in combination with the characteristics of three-dimensional data to study the lesion detection of three-dimensional medical images, which specifically includes:

[0076] like Figure 3 As shown, the method for marking suspicious sites based on three-dimensional medical images of FocalMix includes:

[0077] S1: We propose a generalization of focal loss, a soft training objective that emerges in modern semi-supervised learning frameworks to train detection models.

[0078] The soft target focal loss proposed for semi-supervised learning is defined as:

[0079] SFL(p)=[α0+y(α1-α0)]·|yp| γ CE(y,p)

[0080] Among them, CE(y,p)=-ylog p-(1-y)log(1-p) is the cross entropy loss.

[0081] When y∈{0,1}, the focal loss is a special case of the soft target focal loss we proposed. Based on the focal loss, combining the two terms, the proposed method can be generalized to the case of soft targets, and a soft target focal loss training model is proposed.

[0082] S2: During training, we follow the common practice in computer vision, that is, sampling patches of the same size from the original image. At the same time, we ensure that the edge length of the image patch is divisible by the maximum step size used in FPN. Therefore, each anchor in the image patch can fall into the position of another anchor after rotation or flipping. The extension of each patch is defined as applying these two types of soft transformations on it. In addition, 3D medical images have richer combinations of rotations and flips in different directions. The model is then used to predict the probability of each anchor point in the transformed image patch matching the target. A guessed target for each anchor in the original patch is obtained by inverse transformation (rotation or backward flipping).

[0083] S3: Use hybrid augmentation techniques to make the model linear across training examples for better generalization performance.

[0084] In the embodiment, two hybrid methods suitable for medical image damage detection are studied: image-level hybrid and object-level hybrid.

[0085] The examples are as follows: Figure 4 Image-level blending is shown, with arrows pointing to nodules in the original image, and object-level blending is shown, zooming in on the nodules and positioning them at the center of each image patch for better visualization.

[0086] Embodiment 2

[0087] This embodiment discloses a high-precision intelligent fusion system for mpMRI images and ultrasound images, including:

[0088] The data acquisition module is configured to: acquire mpMRI medical image data and ultrasound image data;

[0089] The image fusion module is configured to: use the ultrasound image data as a reference image, reconstruct the mpMRI medical image data into a three-dimensional model based on the MC algorithm to obtain a floating image; perform initial registration of the floating image and the reference image through affine transformation to obtain an initial position of the floating image;

[0090] Establish a multi-level pyramid decomposition image, starting from the highest level of the pyramid, and calculate the deformation field of the highest level of the pyramid;

[0091] Performing similarity measurement on the initial position of the floating image and the deformation field of the highest layer of the pyramid to obtain a similarity measurement value;

[0092] It is determined whether the similarity measurement value meets the set threshold range. If not, the deformation field is changed and recalculated until the similarity measurement requirement is met. If it is met, the deformation field of the highest layer of the pyramid is interpolated to obtain the deformation field of the bottom of the pyramid; the deformation field of the bottom of the pyramid is applied to the floating image to obtain the final image registration result.

[0093] Embodiment 3

[0094] This embodiment provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps in the high-precision intelligent fusion method of mpMRI images and ultrasound images as described above are implemented.

[0095] Embodiment 4

[0096] This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps in the high-precision intelligent fusion method of mpMRI images and ultrasound images as described above are implemented.

[0097] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) containing computer-usable program code.

[0098] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0099] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0100] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0101] A person skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0102] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A high-precision intelligent fusion method for mpMRI images and ultrasound images, characterized in that: include: Acquire mpMRI medical imaging data and ultrasound imaging data; Using ultrasound image data as reference images, mpMRI medical image data is reconstructed into a three-dimensional model to obtain a floating image; Performing initial registration of the floating image and the reference image through affine transformation to obtain the initial position of the floating image; Establish a multi-level pyramid decomposition image, starting from the highest level of the pyramid, and calculate the deformation field of the highest level of the pyramid; Performing similarity measurement on the initial position of the floating image and the deformation field of the highest layer of the pyramid to obtain a similarity measurement value; Determine whether the similarity measurement value meets a set threshold range. If not, change and recalculate the deformation field until the similarity measurement requirement is met. If so, perform interpolation calculation on the deformation field of the highest layer of the pyramid to obtain the deformation field of the bottom of the pyramid. Apply the deformation field at the bottom of the pyramid to the floating image to obtain the final image registration result; The similarity measurement formula for the initial position of the floating image and the deformation field of the highest level of the pyramid is: In the formula, represents the deformation field, Represents a floating image, represents the reference image, represents the similarity measure component, Represents the regular component.

2. A high-precision intelligent fusion method for mpMRI images and ultrasound images as claimed in claim 1, characterized in that: The method of reconstructing the mpMRI medical image data into a three-dimensional model to obtain a floating image comprises the following steps: Scan the adjacent two layers of the mpMRI original image data layer by layer and construct cubes one by one; Compare the grayscale value of the cube vertex with the threshold of the cube's isosurface to obtain the index value; Build a lookup table and encapsulate the index table in the protocol mapping table; According to the index value, the protocol mapping table points to the corresponding classification edge lookup table to obtain the intersection edge of the current cube that has an intersection with the isosurface; According to the two vertices of the intersecting edge and their normal vectors, the combination of equivalent points constituting the triangle piece in the current cube is determined, and the isosurface is formed by the combination of equivalent points of the triangle pieces in each cube.

3. A high-precision intelligent fusion method for mpMRI images and ultrasound images as claimed in claim 2, characterized in that: The construction of the lookup table is classified and constructed according to the topological structure, each category includes a lookup table, a function pointer and a processing function, each category is processed in a multi-threaded manner, and each category corresponds to a thread.

4. A high-precision intelligent fusion method for mpMRI images and ultrasound images as claimed in claim 2, characterized in that: The coordinates and normal vector of the equivalent point are calculated by the midpoint selection method.

5. A high-precision intelligent fusion method for mpMRI images and ultrasound images as claimed in claim 2, characterized in that: The method for determining the combination of equal-value points constituting a triangle in the current cube is to search the expanded triangle index table through the function pointer in the protocol mapping table according to the data in the edge index table.

6. A high-precision intelligent fusion method for mpMRI images and ultrasound images as claimed in claim 1, characterized in that: The synchronization between multiple threads adopts the critical section method.

7. A high-precision intelligent fusion system for mpMRI images and ultrasound images, characterized in that: include: The data acquisition module is configured to: acquire mpMRI medical image data and ultrasound image data; The image fusion module is configured to: use the ultrasound image data as a reference image, reconstruct the mpMRI medical image data into a three-dimensional model based on the MC algorithm to obtain a floating image; perform initial registration of the floating image and the reference image through affine transformation to obtain an initial position of the floating image; Establish a multi-level pyramid decomposition image, starting from the highest level of the pyramid, and calculate the deformation field of the highest level of the pyramid; Performing similarity measurement on the initial position of the floating image and the deformation field of the highest layer of the pyramid to obtain a similarity measurement value; Determine whether the similarity measurement value meets the set threshold range. If not, change and recalculate the deformation field until the similarity measurement requirement is met. If it is met, interpolate the deformation field of the highest layer of the pyramid to obtain the deformation field of the bottom of the pyramid; apply the deformation field of the bottom of the pyramid to the floating image to obtain the final image registration result. The similarity measurement formula for the initial position of the floating image and the deformation field of the highest level of the pyramid is: In the formula, represents the deformation field, Represents a floating image, represents the reference image, represents the similarity measure component, Represents the regular component.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps in a high-precision intelligent fusion method of mpMRI images and ultrasound images as described in any one of claims 1 to 6 are implemented.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps in the high-precision intelligent fusion method of mpMRI images and ultrasound images as described in any one of claims 1-6 are implemented.

Images