Image processing method and image positioning and guiding system based on multi-modal image fusion

By scoring and contour extraction of multimodal medical images, determining the image type, and accurately positioning, the problem of difficulty in judging after fusion of multimodal medical images is solved, and the accuracy and real-time nature of interventional treatment are improved.

CN119006660AActive Publication Date: 2024-11-22WUHAN BODA BIOTECHNOLOGY CO LTD
View PDF 10 Cites 0 Cited by

Patent Information

Application Number
CN202410994318.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-24
Publication Date
2024-11-22
Estimated Expiration
2044-07-24

AI Technical Summary

Technical Problem

In the prior art, after the multimodal medical image fusion, doctors judge that it is difficult to achieve accurate judgment and treatment guidance due to the diversification of image information.

Method used

By establishing a multimodal medical image set, doctor scoring and data processing, building a sample set, extracting contour information, collecting images in real time and comparing them with the processed images, determining the image type, and guiding treatment based on precise positioning to improve intervention Real-time and accuracy of treatment.

Benefits of technology

The accuracy of image judgment after multimodal medical images fusion and the real-time and accuracy of interventional treatment are improved, ensuring the accuracy of the treatment process.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119006660A_ABST
    Figure CN119006660A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of image fusion, and discloses an image processing method and an image positioning guide system based on multi-modal image fusion. According to the method, multi-modal medical images are fused to obtain fused multi-modal medical images, doctor scoring is carried out, a multi-modal medical image sample set is constructed, data in the constructed multi-modal medical image sample set is processed to obtain the contour of a processed combined medical image, and meanwhile, the contour of the combined medical image is obtained. And for the multi-modal medical image acquired in real time, extracting the contour of the acquired multi-modal medical image in the same way, and comparing the contour of the acquired multi-modal medical image extracted in real time with the contour of the extracted and processed combined medical image to determine the type of the multi-modal medical image acquired in real time. Meanwhile, after the type of the multi-modal medical image collected in real time is determined, accurate positioning between the image and the patient is established, treatment guidance is conducted based on the established positioning, and the real-time performance and accuracy of treatment guidance are improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of image fusion, and in particular to an image processing method and an image positioning guidance system based on multimodal image fusion. Background Art

[0002] With the increase in the types of medical imaging equipment, the fusion of medical images of different modalities collected will provide more diverse information than a single image. However, due to the diversity of image information, it also affects the doctor's judgment to a certain extent.

[0003] There is an existing Chinese patent CN111325703B, which obtains a multimodal image set of the target radiotherapy site, determines the image type to be fused according to the tissue structure type of the target radiotherapy site, selects the image type to be fused for image fusion, and performs operations based on the fused image of the target radiotherapy site. It does not solve the impact on doctors due to the diversity of image information and has certain limitations. Summary of the invention

[0004] 1. Technical issues to be resolved

[0005] In view of the shortcomings of the prior art, the present invention provides an image processing method and an image positioning guidance system based on multimodal image fusion, which have the advantages of real-time and accuracy, and solve the problem that the image after multimodal image fusion affects judgment.

[0006] (II) Technical solution

[0007] In order to solve the technical problem of image impact judgment after multimodal image fusion, the present invention provides the following technical solutions:

[0008] The present invention discloses an image processing method based on multimodal image fusion, which specifically comprises the following steps:

[0009] S1. Establish a multimodal medical image set, and select the image types that need to be fused to perform image fusion, so as to obtain a fused multimodal medical image;

[0010] S2, perform doctor scoring on the fused multimodal medical images, collect doctor scoring data, and construct a multimodal medical image sample set;

[0011] Set the value range of the collected doctor scoring data corresponding to the multimodal medical images to [1, 5];

[0012] Setting "5 points" means that the image quality is excellent and does not affect the judgment;

[0013] Setting "4 points" means that the image quality is good and does not affect the judgment;

[0014] Setting "3 points" means that the image quality is poor and affects the judgment;

[0015] Setting "2 points" indicates that the image quality is poor, which obviously affects the judgment;

[0016] Setting "1 point" means that the image quality is extremely poor and cannot be judged;

[0017] S3, processing the multimodal medical image and the corresponding doctor scoring data after fusion in the constructed multimodal medical image sample set to obtain the outline of the processed combined medical image;

[0018] S4, using an image acquisition device, collecting and processing a multimodal medical image of the patient's affected part in real time to obtain a contour of the multimodal medical image of the patient's affected part collected in real time;

[0019] S5. Compare the contour of the multimodal medical image of the patient's affected part acquired in real time with the contour of the processed combined medical image to determine the type of the multimodal medical image of the patient's affected part acquired in real time.

[0020] The present invention obtains a fused multimodal medical image by fusing multimodal medical images, performs doctor scoring, and constructs a multimodal medical image sample set. At the same time, the fused multimodal medical image and the corresponding doctor scoring data in the constructed multimodal medical image sample set are processed to obtain the contour of the processed combined medical image. At the same time, the contour of the multimodal medical image of the patient's affected part collected in real time is extracted in the same way, and the contour of the multimodal medical image of the patient's affected part extracted in real time is compared with the contour of the extracted and processed combined medical image to determine the type of the multimodal medical image of the patient's affected part collected in real time.

[0021] Preferably, the method further includes S6: after determining the type of multimodal medical image acquired in real time, establishing precise positioning between the image and the patient, and guiding treatment based on the established positioning.

[0022] After determining the type of multimodal medical images acquired in real time, precise positioning between the image and the patient is established, and treatment guidance is performed based on the established positioning, thereby improving the real-time and accuracy of interventional treatment guidance.

[0023] Preferably, the processing of the fused multimodal medical images and the corresponding doctor scoring data in the constructed multimodal medical image sample set comprises the following steps:

[0024] S31, processing the doctor scoring data corresponding to the fused multimodal medical image;

[0025] S32, performing contour extraction on the fused multimodal medical image to obtain the contour of the processed combined medical image.

[0026] Preferably, the processing of the doctor scoring data comprises the following steps:

[0027] S311, summarizing the doctor scoring data corresponding to each fused multimodal medical image, and calculating the subjective score of each fused multimodal medical image;

[0028] The subjective score calculation formula is as follows:

[0029]

[0030] in, represents the calculated subjective score of the fused multimodal medical image, u i represents the score data of the i-th doctor, and T represents the number of doctors;

[0031] Setting a subjective score threshold of the fused multimodal medical image, and removing the fused multimodal medical image with a subjective score lower than the set subjective score threshold;

[0032] S312, calculating an objective evaluation score of each fused multimodal medical image based on the subjective score of each fused multimodal medical image;

[0033] The standard deviation of the subjective score is calculated based on the subjective score of each fused multimodal medical image. The calculation formula is as follows:

[0034]

[0035] Where σ represents the standard deviation of the calculated subjective scores;

[0036] The correlation of the doctor rating data was calculated based on the Pearson correlation coefficient, and the calculation formula is as follows:

[0037]

[0038] Among them, cov(X,Y) represents the covariance between the doctor X score data and the doctor Y score data, σ X represents the standard deviation of doctor X's score data, σ Y represents the standard deviation of doctor Y score data, ρ X,Y Represents the correlation between doctor X's score data and doctor Y's score data;

[0039] A correlation threshold between the doctor scoring data is set, and the fused multimodal medical images below the set threshold are removed.

[0040] Preferably, the step of extracting contours from the fused multimodal medical images to obtain contours of the processed combined medical image comprises the following steps:

[0041] S321, performing image binarization processing on the fused multimodal medical image;

[0042] The initial grayscale threshold k is selected to classify all pixels in the multimodal medical image into two categories C1 and C2;

[0043] Set C1 to be the pixel category that is less than or equal to the grayscale threshold k, and C2 to be the pixel category that is greater than the grayscale threshold k;

[0044] Set the grayscale mean of pixel category C1 to h1, the grayscale mean of pixel category C2 to h2, and the global grayscale mean to h3;

[0045] The probability that a pixel in the fused multimodal medical image belongs to pixel category C1 is set to p1, and the probability that a pixel belongs to pixel category C2 is set to p2;

[0046] The binarization formula is as follows:

[0047] h3=h1×p1+h2×p2;

[0048]

[0049] Wherein, η represents the binarization threshold;

[0050] The grayscale values ​​greater than the binarization threshold are set to 255, and the grayscale values ​​less than or equal to the binarization threshold are set to 0;

[0051] Aggregating pixels in the binarized and fused multimodal medical image to obtain a binarized and fused multimodal medical image;

[0052] S322, performing contour extraction on the fused multimodal medical image after binarization processing;

[0053] Establish a one-dimensional array to record the grayscale values ​​of the eight neighborhoods around each pixel of the multimodal medical image after binarization. When the grayscale values ​​of the eight neighborhoods around the pixel are the same as the grayscale value of the center point, the pixel is set to be inside the object and deleted.

[0054] When the grayscale values ​​of the eight neighborhoods around a pixel are different from the grayscale value of the center point, the pixel is set to be at the edge of the object and is retained;

[0055] Each pixel point in the fused multimodal medical image after binarization processing is traversed and summarized to obtain the contour of the processed combined medical image.

[0056] The present invention calculates the objective score of each fused multimodal medical image by processing the doctor's scoring data of each fused multimodal medical image corresponding to the multimodal medical image, and processes the fused multimodal medical image. At the same time, for the processed fused multimodal medical image, the contour of the processed fused multimodal medical image is extracted by contour extraction, thereby improving the accuracy of interventional treatment guidance.

[0057] Preferably, comparing the contour of the multimodal medical image of the affected part of the patient acquired in real time with the contour of the processed combined medical image to determine the type of the multimodal medical image of the affected part of the patient acquired in real time comprises the following steps:

[0058] S51, matching the contour of the multimodal medical image of the patient's affected part acquired in real time with the contour of the processed combined medical image;

[0059] S52: Determine the type of the multimodal medical image of the patient's affected part acquired in real time according to the overlap degree of the contour lines.

[0060] Preferably, determining the type of the multimodal medical image of the patient's affected part acquired in real time according to the overlap degree of the contour lines comprises the following steps:

[0061] Set the contour of the processed combined medical image to be L m , the multimodal medical image of the patient's affected part collected in real time is D, and the contour of the multimodal medical image of the patient's affected part collected in real time is L D ; Use the similarity function S(Θ) to judge the contour image L m , L D degree of overlap;

[0062]

[0063] Among them, S(Θ) represents the contour image L m , L D The similarity of the two contours L m With L D The area of ​​the overlapped part, A(L m ) and A(L D ) are the areas of the corresponding contours; 0≤S(Θ)≤1, the larger S(Θ), the larger L m and L D The more similar, the more L m and L D When they overlap, S(Θ)=1;

[0064] Set the similarity threshold, when L m and L D When the similarity is greater than the set threshold, it means L m and LD The type is consistent.

[0065] The present invention matches the contours of multimodal medical images of the patient's affected part collected in real time with the contours of the processed medical images, calculates the similarity between the two groups of contour images by contour comparison, and judges the type of the multimodal medical images of the patient's affected part collected in real time based on the calculated similarity, thereby ensuring the accuracy of interventional treatment guidance.

[0066] Preferably, after determining the type of multimodal medical image acquired in real time, establishing accurate positioning between the image and the patient, and guiding treatment based on the established positioning includes the following steps:

[0067] S61, measuring the position and posture of the patient, and establishing a three-dimensional coordinate system of the position and posture of the patient;

[0068] S62. Based on the established three-dimensional coordinate system of the patient's position and posture, the contour of the multimodal medical image of the patient's affected part acquired in real time after the type is determined is projected into the established three-dimensional coordinate system, and the relationship between the projection and the three-dimensional coordinate system is calculated.

[0069] S63. Based on the relationship between the calculated projection and the three-dimensional coordinate system, real-time acquisition, real-time projection, and real-time positioning guidance of the affected part are performed.

[0070] Preferably, the calculating the relationship between the projection and the three-dimensional coordinate system comprises the following steps:

[0071] The relationship between the calculated projection and the three-dimensional coordinate system is shown below;

[0072] Assume Q is the coordinate of the affected part in the three-dimensional coordinate system, and q is the coordinate of the contour of the multimodal medical image of the affected part of the patient acquired in real time in the three-dimensional coordinate system;

[0073] Q=(x Q ,y Q ,z Q ,w Q );

[0074] q=(x q ,y q ,w q );

[0075] Among them, x Q ,y Q ,z Q Respectively represent the horizontal, vertical and vertical coordinates of the affected area in the three-dimensional coordinate system; q ,y q They represent the horizontal and vertical coordinates of the contour of the multimodal medical image of the patient's affected part collected in real time in the three-dimensional coordinate system, w QIndicates the wheelbase parameter of the established three-dimensional coordinate system, w q represents the axis distance parameter of the multimodal medical image of the patient's affected part acquired in real time;

[0076] Set when w q =w Q When the multi-modal medical image of the patient's affected part is collected in real time, the contour projection is coaxial with the established three-dimensional coordinate system;

[0077] When the contour projection of the multimodal medical image of the patient's affected part acquired in real time is coaxial with the established three-dimensional coordinate system, the relationship between the two sets of coordinates is as follows:

[0078]

[0079] Among them, f x represents the focal length of the camera on the x-axis, f y represents the camera focal length on the y axis, e x represents the horizontal coordinate of the center point of the affected area on the x-axis, e y It represents the ordinate of the center point of the affected area on the y-axis; represents the rotation vector matrix, r 11 Represents the rotation vector of the first row and first column in the rotation vector matrix, Represents the translation vector matrix, and t1 represents the translation vector of the first row.

[0080] The present invention establishes a three-dimensional coordinate system of the patient's position and posture by measuring the patient's position and posture, and calculates the relationship between the projection and the three-dimensional coordinate system by projecting the contours of the multimodal medical image collected in real time, completes the conversion between the projection and the three-dimensional coordinate system, and realizes precise positioning of the affected area, thereby improving the real-time and accuracy of interventional treatment guidance.

[0081] The present invention discloses an image positioning guidance system based on multimodal image fusion, comprising: a fusion scoring module, a data processing module, a contour extraction module, a type determination module and a precise positioning module;

[0082] The fusion scoring module is used to fuse the multimodal medical images, obtain the fused multimodal medical images and perform doctor scoring;

[0083] The data processing module is used to process the collected multimodal medical images and the doctor scoring data of the multimodal medical images;

[0084] The contour extraction module is used to extract contours of the processed multimodal medical image;

[0085] The type determination module is used to determine the type of the multimodal medical image acquired in real time by contour comparison;

[0086] The precise positioning module is used to precisely position the affected area by means of three-dimensional coordinate conversion and real-time acquisition and projection.

[0087] (III) Beneficial effects

[0088] Compared with the prior art, it has the following beneficial effects:

[0089] 1. The present invention fuses multimodal medical images to obtain a fused multimodal medical image and performs doctor scoring, and constructs a multimodal medical image sample set. At the same time, the fused multimodal medical image and the corresponding doctor scoring data in the constructed multimodal medical image sample set are processed to obtain the contour of the processed combined medical image. At the same time, the contour of the multimodal medical image of the patient's affected part collected in real time is extracted in the same way for the multimodal medical image of the patient's affected part collected in real time, and the contour of the multimodal medical image of the patient's affected part extracted in real time is compared with the contour of the processed combined medical image to determine the type of the multimodal medical image of the patient's affected part collected in real time. After determining the type of the multimodal medical image of the patient's affected part collected in real time, accurate positioning is established between the image and the patient, and treatment guidance is performed based on the established positioning, thereby improving the real-time and accuracy of interventional treatment guidance.

[0090] 2. The present invention calculates the objective score of each fused multimodal medical image by processing the doctor's scoring data of each fused multimodal medical image corresponding to the multimodal medical image, and processes the fused multimodal medical image. At the same time, for the processed fused multimodal medical image, the contour of the processed fused multimodal medical image is extracted by contour extraction, thereby improving the accuracy of interventional treatment guidance.

[0091] 3. The present invention matches the contours of the multimodal medical images of the patient's affected part collected in real time with the contours of the processed medical images, calculates the similarity between the two sets of contour images by contour comparison, and determines the type of the multimodal medical images of the patient's affected part collected in real time based on the calculated similarity, thereby ensuring the accuracy of interventional treatment guidance.

[0092] 4. The invention establishes a three-dimensional coordinate system of the patient's position and posture by measuring the patient's position and posture, and calculates the relationship between the projection and the three-dimensional coordinate system by projecting the contours of the multimodal medical images collected in real time, completes the conversion between the projection and the three-dimensional coordinate system, and realizes precise positioning of the affected area, thereby improving the real-time and accuracy of interventional treatment guidance. BRIEF DESCRIPTION OF THE DRAWINGS

[0093] Figure 1 The figure is a schematic diagram of the process structure of an image positioning and guiding system based on multimodal image fusion of the present invention. DETAILED DESCRIPTION

[0094] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0095] Example 1

[0096] This embodiment discloses an image processing method based on multimodal image fusion, which specifically includes the following steps:

[0097] S1. Establish a multimodal medical image set, and select the image types that need to be fused to perform image fusion, so as to obtain a fused multimodal medical image;

[0098] Selecting the image type to be fused for image fusion includes the following steps:

[0099] Image fusion based on pixel weighted average method;

[0100] Set the weight of each image type;

[0101] Select image types with the same image size for fusion;

[0102] The fused image formula is as follows:

[0103] F(α,β)=ω1B(α,β)+ω2E(α,β);

[0104] Among them, F(α, β) represents the pixel at (α, β) in the fused image, B(α, β) represents the pixel at (α, β) in image B, E(α, β) represents the pixel at (α, β) in image E, and ω1 and ω2 represent the weights of the corresponding image types.

[0105] S2, perform doctor scoring on the fused multimodal medical images, collect doctor scoring data, and construct a multimodal medical image sample set;

[0106] Set the value range of the collected doctor scoring data corresponding to the multimodal medical images to [1, 5];

[0107] Setting "5 points" means that the image quality is excellent and does not affect the judgment;

[0108] Setting "4 points" means that the image quality is good and does not affect the judgment;

[0109] Setting "3 points" means that the image quality is poor and affects the judgment;

[0110] Setting "2 points" indicates that the image quality is poor, which obviously affects the judgment;

[0111] Setting "1 point" means that the image quality is extremely poor and cannot be judged;

[0112] S3, processing the multimodal medical image and the corresponding doctor scoring data after fusion in the constructed multimodal medical image sample set to obtain the outline of the processed combined medical image;

[0113] Processing the multimodal medical images in the multimodal medical image sample set and the doctor scoring data corresponding to the multimodal medical images includes the following steps:

[0114] S31, processing the doctor scoring data corresponding to the fused multimodal medical image;

[0115] S311, summarizing the doctor scoring data corresponding to each fused multimodal medical image, and calculating the subjective score of each fused multimodal medical image;

[0116] The subjective score calculation formula is as follows:

[0117]

[0118] in, represents the calculated subjective score of the fused multimodal medical image, u i represents the score data of the i-th doctor, and T represents the number of doctors;

[0119] Setting a subjective score threshold of the fused multimodal medical image, and removing the fused multimodal medical image with a subjective score lower than the set subjective score threshold;

[0120] S312, calculating an objective evaluation score of each fused multimodal medical image based on the subjective score of each fused multimodal medical image;

[0121] The standard deviation of the subjective score is calculated based on the subjective score of each fused multimodal medical image. The calculation formula is as follows:

[0122]

[0123] Where σ represents the standard deviation of the calculated subjective scores;

[0124] The correlation of the doctor rating data was calculated based on the Pearson correlation coefficient, and the calculation formula is as follows:

[0125]

[0126] Among them, cov(X,Y) represents the covariance between the doctor X score data and the doctor Y score data, σ X represents the standard deviation of doctor X's score data, σ Y represents the standard deviation of doctor Y score data, ρ X,Y Represents the correlation between doctor X's score data and doctor Y's score data;

[0127] A correlation threshold between doctor scoring data is set, and fused multimodal medical images with values ​​below the set threshold are removed;

[0128] S32, performing contour extraction on the fused multimodal medical image to obtain the contour of the processed combined medical image;

[0129] S321, performing image binarization processing on the fused multimodal medical image;

[0130] The initial grayscale threshold k is selected to classify all pixels in the multimodal medical image into two categories C1 and C2;

[0131] Set C1 to be the pixel category that is less than or equal to the grayscale threshold k, and C2 to be the pixel category that is greater than the grayscale threshold k;

[0132] Set the grayscale mean of pixel category C1 to h1, the grayscale mean of pixel category C2 to h2, and the global grayscale mean to h3;

[0133] The probability that a pixel in the fused multimodal medical image belongs to pixel category C1 is set to p1, and the probability that a pixel belongs to pixel category C2 is set to p2;

[0134] The binarization formula is as follows:

[0135] h3=h1×p1+h2×p2;

[0136]

[0137] Wherein, η represents the binarization threshold;

[0138] The grayscale values ​​greater than the binarization threshold are set to 255, and the grayscale values ​​less than or equal to the binarization threshold are set to 0;

[0139] Aggregating pixels in the binarized and fused multimodal medical image to obtain a binarized and fused multimodal medical image;

[0140] S322, performing contour extraction on the fused multimodal medical image after binarization processing;

[0141] Establish a one-dimensional array to record the grayscale values ​​of the eight neighborhoods around each pixel of the multimodal medical image after binarization. When the grayscale values ​​of the eight neighborhoods around the pixel are the same as the grayscale value of the center point, the pixel is set to be inside the object and deleted.

[0142] When the grayscale values ​​of the eight neighborhoods around a pixel are different from the grayscale value of the center point, the pixel is set to be at the edge of the object and is retained;

[0143] Traversing and summarizing each pixel point in the fused multimodal medical image after binarization processing to obtain the contour of the processed combined medical image;

[0144] S4, using an image acquisition device, collecting and processing a multimodal medical image of the patient's affected part in real time to obtain a contour of the multimodal medical image of the patient's affected part collected in real time;

[0145] S5, comparing the contour of the multimodal medical image of the patient's affected part acquired in real time with the contour of the processed combined medical image to determine the type of the multimodal medical image of the patient's affected part acquired in real time;

[0146] Comparing the contour of the multimodal medical image of the affected part of the patient acquired in real time with the contour of the processed combined medical image to determine the type of the multimodal medical image of the affected part of the patient acquired in real time includes the following steps:

[0147] S51, matching the contour of the multimodal medical image of the patient's affected part extracted in real time with the contour of the processed medical image;

[0148] S52, determining the type of the multimodal medical image of the patient's affected part acquired in real time according to the overlap degree of the contour lines;

[0149] Set the contour of the processed combined medical image to be L m , the multimodal medical image of the patient's affected part collected in real time is D, and the contour of the multimodal medical image of the patient's affected part collected in real time is L D ; Use the similarity function S(Θ) to judge the contour image L m , L D degree of overlap;

[0150]

[0151] Among them, S(Θ) represents the contour image L m , L D The similarity of the two contours L m With L D The area of ​​the overlapped part, A(L m ) and A(L D) are the areas of the corresponding contours; 0≤S(Θ)≤1, the larger S(Θ), the larger L m and L D The more similar, the more L m and L D When they overlap, S(Θ)=1;

[0152] Set the similarity threshold, when L m and L D When the similarity is greater than the set threshold, it means L m and L D The type is consistent.

[0153] Furthermore, in some embodiments, it further includes S6: after determining the type of the multimodal medical image acquired in real time, establishing a precise positioning between the image and the patient, and guiding treatment based on the established positioning;

[0154] After determining the type of multimodal medical image to be acquired in real time, establishing accurate positioning between the image and the patient, and guiding treatment based on the established positioning include the following steps:

[0155] S61, measuring the position and posture of the patient, and establishing a three-dimensional coordinate system of the position and posture of the patient;

[0156] S62, based on the established three-dimensional coordinate system of the patient's position and posture, projecting the contour of the multimodal medical image acquired in real time after the type is determined into the established three-dimensional coordinate system, and calculating the relationship between the projection and the three-dimensional coordinate system;

[0157] The relationship between the calculated projection and the three-dimensional coordinate system is shown below;

[0158] Assume Q is the coordinate of the affected part in the three-dimensional coordinate system, and q is the coordinate of the contour of the multimodal medical image of the affected part of the patient acquired in real time in the three-dimensional coordinate system;

[0159] Q=(x Q ,y Q ,z Q ,w Q );

[0160] q=(x q ,y q ,w q );

[0161] Among them, x Q ,y Q ,z Q Respectively represent the horizontal, vertical and vertical coordinates of the affected area in the three-dimensional coordinate system; q ,y q They represent the horizontal and vertical coordinates of the contour of the multimodal medical image of the patient's affected part collected in real time in the three-dimensional coordinate system, w QIndicates the wheelbase parameter of the established three-dimensional coordinate system, w q represents the axis distance parameter of the multimodal medical image of the patient's affected part acquired in real time;

[0162] Set when w q =w Q When the multi-modal medical image of the patient's affected part is collected in real time, the contour projection is coaxial with the established three-dimensional coordinate system;

[0163] When the contour projection of the multimodal medical image of the patient's affected part acquired in real time is coaxial with the established three-dimensional coordinate system, the relationship between the two sets of coordinates is as follows:

[0164]

[0165] Among them, f x represents the focal length of the camera on the x-axis, f y represents the camera focal length on the y axis, e x represents the horizontal coordinate of the center point of the affected area on the x-axis, e y It represents the ordinate of the center point of the affected area on the y-axis; represents the rotation vector matrix, r 11 Represents the rotation vector of the first row and first column in the rotation vector matrix, represents the translation vector matrix, t1 represents the translation vector of the first row;

[0166] S63, based on the relationship between the calculated projection and the three-dimensional coordinate system, real-time acquisition and real-time projection of the affected part are performed to complete real-time guidance of treatment of the affected part;

[0167] Example 2

[0168] This embodiment discloses an image positioning guidance system based on multimodal image fusion, including: a fusion scoring module, a data processing module, a contour extraction module, a type determination module and a precise positioning module;

[0169] The fusion scoring module is used to fuse the multimodal medical images, obtain the fused multimodal medical images and perform doctor scoring;

[0170] The data processing module is used to process the collected multimodal medical images and the doctor scoring data of the multimodal medical images;

[0171] The contour extraction module is used to extract contours of the processed multimodal medical image;

[0172] The type determination module is used to determine the type of the multimodal medical image acquired in real time by contour comparison;

[0173] The precise positioning module is used to precisely position the affected area by means of three-dimensional coordinate conversion and real-time acquisition and projection.

[0174] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the invention, and the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. An image processing method based on multimodal image fusion, characterized in that: The following steps are involved: S1. Establish a multimodal medical image set, and select the image types that need to be fused to perform image fusion, so as to obtain a fused multimodal medical image; S2, perform doctor scoring on the fused multimodal medical images, collect doctor scoring data, and construct a multimodal medical image sample set; S3, processing the multimodal medical image and the corresponding doctor scoring data after fusion in the constructed multimodal medical image sample set to obtain the outline of the processed combined medical image; S4, using an image acquisition device, collecting and processing a multimodal medical image of the patient's affected part in real time to obtain a contour of the multimodal medical image of the patient's affected part collected in real time; S5. Compare the contour of the multimodal medical image of the patient's affected part acquired in real time with the contour of the processed combined medical image to determine the type of the multimodal medical image of the patient's affected part acquired in real time.

2. The image processing method based on multimodal image fusion according to claim 1, characterized in that: The step of selecting the image type to be fused for image fusion comprises the following steps: Image fusion based on pixel weighted average method; Set the weight of each image type; Select image types with the same image size for fusion; The fused image formula is as follows: F(α,β)=ω1B(α,β)+ω2E(α,β); Among them, F(α, β) represents the pixel at (α, β) in the fused image, B(α, β) represents the pixel at (α, β) in image B, E(α, β) represents the pixel at (α, β) in image E, and ω1 and ω2 represent the weights of the corresponding image types.

3. The image processing method based on multimodal image fusion according to claim 1, characterized in that: The processing of the fused multimodal medical images and the corresponding doctor scoring data in the constructed multimodal medical image sample set comprises the following steps: S31, processing the doctor scoring data corresponding to the fused multimodal medical image; S32, performing contour extraction on the fused multimodal medical image to obtain the contour of the processed combined medical image.

4. The image processing method based on multimodal image fusion according to claim 3, characterized in that: The processing of the doctor scoring data corresponding to the fused multimodal medical image comprises the following steps: S311, summarizing the doctor scoring data corresponding to each fused multimodal medical image, and calculating the subjective score of each fused multimodal medical image; The subjective score calculation formula is as follows: in, represents the calculated subjective score of the fused multimodal medical image, u i represents the score data of the i-th doctor, and T represents the number of doctors; Setting a subjective score threshold of the fused multimodal medical image, and removing the fused multimodal medical image with a subjective score lower than the set subjective score threshold; S312, calculating an objective evaluation score of each fused multimodal medical image based on the subjective score of each fused multimodal medical image; The standard deviation of the subjective score is calculated based on the subjective score of each fused multimodal medical image. The calculation formula is as follows: Where σ represents the standard deviation of the calculated subjective scores; The correlation of the doctor rating data was calculated based on the Pearson correlation coefficient, and the calculation formula is as follows: Among them, cov(X,Y) represents the covariance between the doctor X score data and the doctor Y score data, σ X represents the standard deviation of doctor X's score data, σ Y represents the standard deviation of doctor Y score data, ρ X,Y Represents the correlation between doctor X's score data and doctor Y's score data; A correlation threshold between the doctor scoring data is set, and the fused multimodal medical images below the set threshold are removed.

5. The image processing method based on multimodal image fusion according to claim 3, characterized in that: The step of extracting the contour of the fused multimodal medical image to obtain the contour of the processed combined medical image comprises the following steps: S321, performing image binarization processing on the fused multimodal medical image; The initial grayscale threshold k is selected to classify all pixels in the multimodal medical image into two categories C1 and C2; Set C1 to be the pixel category that is less than or equal to the grayscale threshold k, and C2 to be the pixel category that is greater than the grayscale threshold k; Set the grayscale mean of pixel category C1 to h1, the grayscale mean of pixel category C2 to h2, and the global grayscale mean to h3; The probability that a pixel in the fused multimodal medical image belongs to pixel category C1 is set to p1, and the probability that a pixel belongs to pixel category C2 is set to p2; The binarization formula is as follows: h3=h1×p1+h2×p2; Wherein, η represents the binarization threshold; The grayscale values ​​greater than the binarization threshold are set to 255, and the grayscale values ​​less than or equal to the binarization threshold are set to 0; Aggregating pixels in the binarized and fused multimodal medical image to obtain a binarized and fused multimodal medical image; S322, performing contour extraction on the fused multimodal medical image after the binarization process to obtain the contour of the processed combined medical image; A one-dimensional array is established to record the grayscale values ​​of the eight neighborhoods around each pixel of the fused multimodal medical image after binarization. When the grayscale values ​​of the eight neighborhoods around the pixel are the same as the grayscale value of the center point, the pixel is set to be inside the object and deleted. When the grayscale values ​​of the eight neighborhoods around a pixel are different from the grayscale value of the center point, the pixel is set to be at the edge of the object and is retained; Each pixel point in the fused multimodal medical image after binarization processing is traversed and summarized to obtain the contour of the processed combined medical image.

6. The image processing method based on multimodal image fusion according to claim 1, characterized in that: The step of comparing the contour of the multimodal medical image of the affected part of the patient acquired in real time with the contour of the processed combined medical image to determine the type of the multimodal medical image of the affected part of the patient acquired in real time comprises the following steps: S51, matching the contour of the multimodal medical image of the patient's affected part acquired in real time with the contour of the processed combined medical image; S52: Determine the type of the multimodal medical image of the patient's affected part acquired in real time according to the overlap degree of the contour lines.

7. The image processing method based on multimodal image fusion according to claim 6, characterized in that: Determining the type of the multimodal medical image of the patient's affected part acquired in real time according to the overlap degree of the contour lines comprises the following steps: Set the contour of the processed combined medical image to be L m , the multimodal medical image of the patient's affected part collected in real time is D, and the contour of the multimodal medical image of the patient's affected part collected in real time is L D ; Use the similarity function S(Θ) to judge the contour image L m , L D degree of overlap; Among them, S(Θ) represents the contour image L m , L D The similarity of the two contours L m With L D The area of ​​the overlapped part, A(L m ) and A(L D ) are the areas of the corresponding contours; 0≤S(Θ)≤1, the larger S(Θ), the larger L m and L D The more similar, the more L m and L D When they overlap, S(Θ)=1; Set the similarity threshold, when L m and L D When the similarity is greater than the set threshold, it means L m and L D The type is consistent.

8. The image processing method based on multimodal image fusion according to claim 1, characterized in that: The method further includes S6, after determining the type of the multimodal medical image of the patient's affected part acquired in real time, establishing accurate positioning between the image and the patient, including the following steps: S61, measuring the position and posture of the patient, and establishing a three-dimensional coordinate system of the position and posture of the patient; S62, based on the established three-dimensional coordinate system of the patient's position and posture, projecting the contour of the multimodal medical image of the patient's affected part acquired in real time after the type is determined into the established three-dimensional coordinate system, and calculating the relationship between the projection and the three-dimensional coordinate system; S63. Based on the relationship between the calculated projection and the three-dimensional coordinate system, real-time acquisition, real-time projection, and real-time positioning guidance of the affected part are performed.

9. The image processing method based on multimodal image fusion according to claim 8, characterized in that: The calculation of the relationship between the projection and the three-dimensional coordinate system comprises the following steps: Assume Q is the coordinate of the affected part in the three-dimensional coordinate system, and q is the coordinate of the contour of the multimodal medical image of the affected part of the patient acquired in real time in the three-dimensional coordinate system; Q=(x Q ,y Q ,z Q ,w Q ); q=(x q ,y q ,w q ); Among them, x Q ,y Q ,z Q Respectively represent the horizontal, vertical and vertical coordinates of the affected area in the three-dimensional coordinate system; q ,y q They represent the horizontal and vertical coordinates of the contour of the multimodal medical image of the patient's affected part collected in real time in the three-dimensional coordinate system, w Q Indicates the wheelbase parameter of the established three-dimensional coordinate system, w q represents the axis distance parameter of the multimodal medical image of the patient's affected part acquired in real time; Set when w q =w Q When the multi-modal medical image of the patient's affected part is collected in real time, the contour projection is coaxial with the established three-dimensional coordinate system; When the contour projection of the multimodal medical image of the patient's affected part acquired in real time is coaxial with the established three-dimensional coordinate system, the relationship between the two sets of coordinates is as follows: Among them, f x represents the focal length of the camera on the x-axis, f y represents the y-axis camera focal length, e x represents the horizontal coordinate of the center point of the affected area on the x-axis, e y It represents the ordinate of the center point of the affected area on the y-axis; represents the rotation vector matrix, r 11 Represents the rotation vector of the first row and first column in the rotation vector matrix, Represents the translation vector matrix, and t1 represents the translation vector of the first row.

10. An image positioning and guiding system for implementing the image processing method based on multimodal image fusion according to any one of claims 1 to 9, characterized in that: include: Fusion scoring module, data processing module, contour extraction module, type determination module and precise positioning module; The fusion scoring module is used to fuse the multimodal medical images, obtain the fused multimodal medical images and perform doctor scoring; The data processing module is used to process the fused multimodal medical images and the corresponding doctor scoring data; The contour extraction module is used to extract contours of the processed multimodal medical image; The type determination module is used to determine the type of the multimodal medical image of the patient's affected part acquired in real time by means of contour comparison; The precise positioning module is used to precisely position the affected area by means of three-dimensional coordinate conversion and real-time acquisition and projection.

Citation Information

Patent Citations

  • An unsupervised segmentation method for multi-mode brain tumor MRI

    CN109671054A

  • Sample library construction method based on automatic identification of deep-sea macrobenthos

    CN110674823A

  • Novel multi-modal fusion auxiliary diagnosis method based on rectal cancer imaging omics research

    CN111599464A

  • Multi-modal medical image registration fusion method and device and electronic equipment

    CN113450294A

  • Multi-modal medical image fusion method and device and terminal equipment

    CN114494182A