Image processing method and image positioning guidance system based on multimodal image fusion

Abstract

The present invention relates to the technical field of image fusion, and discloses an image processing method and an image positioning guidance system based on multimodal image fusion. The method fuses multimodal medical images to obtain a fused multimodal medical image, performs doctor scoring, constructs a multimodal medical image sample set, and processes the data in the constructed multimodal medical image sample set to obtain the contour of the processed combined medical image. Simultaneously, the contour of the multimodal medical image acquired in real time is extracted using the same method, and the contour of the real-time extracted multimodal medical image is compared with the contour of the extracted and processed combined medical image to determine the type of the real-time acquired multimodal medical image. After determining the type of the real-time acquired multimodal medical image, precise positioning is established between the image and the patient, and treatment guidance is performed based on the established positioning, thereby improving the real-time and accuracy of the treatment guidance.

Description

Technical Field

[0001] The present invention relates to the technical field of image fusion, and in particular to an image processing method and an image positioning guidance system based on multimodal image fusion. Background Art

[0002] With the increasing variety of medical imaging devices, the fusion of medical images collected from different modalities will provide more diverse information than a single image. However, this diversity of image information also affects doctors' judgment to a certain extent.

[0003] There is an existing Chinese patent CN111325703B. This patent obtains a multimodal image set of the target radiotherapy site, determines the image type to be fused according to the tissue structure type of the target radiotherapy site, selects the image type to be fused for image fusion, and performs operations based on the fused image of the target radiotherapy site. It does not solve the impact on doctors due to the diversity of image information and has certain limitations. Summary of the Invention

[0004] (1) Technical problems solved

[0005] In response to the shortcomings of the existing technology, the present invention provides an image processing method and image positioning guidance system based on multimodal image fusion, which has the advantages of real-time and accuracy, and solves the problem of image impact judgment after multimodal image fusion.

[0006] (2) Technical solution

[0007] To solve the above-mentioned technical problem of image impact judgment after multimodal image fusion, the present invention provides the following technical solutions:

[0008] The present invention discloses an image processing method based on multimodal image fusion, which specifically includes the following steps:

[0009] S1. Establish a multimodal medical image set and select the image types that need to be fused to perform image fusion to obtain a fused multimodal medical image;

[0010] S2. Perform doctor ratings on the fused multimodal medical images, collect doctor rating data, and construct a multimodal medical image sample set;

[0011] Set the value range of the collected doctor scoring data corresponding to the multimodal medical images to [1, 5];

[0012] Setting "5 points" means that the image quality is excellent and does not affect the judgment;

[0013] Setting "4 points" means that the image quality is good and does not affect the judgment;

[0014] Setting "3 points" indicates that the image quality is poor and affects the judgment;

[0015] Setting "2 points" indicates poor image quality, which significantly affects judgment;

[0016] Setting "1 point" means that the image quality is extremely poor and cannot be judged;

[0017] S3, processing the multimodal medical image and the corresponding doctor scoring data after fusion in the constructed multimodal medical image sample set to obtain the outline of the processed combined medical image;

[0018] S4. Using an image acquisition device, collect and process multimodal medical images of the patient's affected area in real time to obtain a contour of the multimodal medical image of the patient's affected area collected in real time;

[0019] S5. Compare the contour of the multimodal medical image of the patient's affected part acquired in real time with the contour of the processed combined medical image to determine the type of the multimodal medical image of the patient's affected part acquired in real time.

[0020] The present invention fuses multimodal medical images to obtain a fused multimodal medical image and performs doctor scoring, and constructs a multimodal medical image sample set. The fused multimodal medical image and the corresponding doctor scoring data in the constructed multimodal medical image sample set are processed to obtain the contour of the processed combined medical image. At the same time, the contour of the multimodal medical image of the patient's affected part collected in real time is extracted in the same manner, and the contour of the multimodal medical image of the patient's affected part extracted in real time is compared with the contour of the extracted and processed combined medical image to determine the type of the multimodal medical image of the patient's affected part collected in real time.

[0021] Preferably, the method further includes S6: after determining the type of the multimodal medical image acquired in real time, establishing precise positioning between the image and the patient, and guiding treatment based on the established positioning.

[0022] After determining the type of multimodal medical images collected in real time, precise positioning between the image and the patient is established, and treatment guidance is performed based on the established positioning, thereby improving the real-time and accuracy of interventional treatment guidance.

[0023] Preferably, the processing of the fused multimodal medical images and the corresponding doctor scoring data in the constructed multimodal medical image sample set comprises the following steps:

[0024] S31, processing the doctor scoring data corresponding to the fused multimodal medical image;

[0025] S32. Perform contour extraction on the fused multimodal medical image to obtain the contour of the processed combined medical image.

[0026] Preferably, the processing of the doctor scoring data comprises the following steps:

[0027] S311, summarizing the doctor rating data corresponding to each fused multimodal medical image, and calculating the subjective score of each fused multimodal medical image;

[0028] The formula for calculating the subjective score is as follows:

[0029]

[0030] in, represents the calculated subjective score of the fused multimodal medical image, u i represents the scoring data of the i-th doctor, and T represents the number of doctors;

[0031] Setting a subjective score threshold of the fused multimodal medical image, and removing the fused multimodal medical image with a subjective score lower than the set subjective score threshold;

[0032] S312. Calculating an objective evaluation score for each fused multimodal medical image based on the subjective score of each fused multimodal medical image;

[0033] The standard deviation of the subjective scores is calculated based on the subjective scores of each fused multimodal medical image. The calculation formula is as follows:

[0034]

[0035] Where σ represents the standard deviation of the calculated subjective scores;

[0036] The correlation of doctor rating data was calculated based on the Pearson correlation coefficient. The calculation formula is as follows:

[0037]

[0038] Among them, cov(X,Y) represents the covariance between the doctor X score data and the doctor Y score data, σ X represents the standard deviation of doctor X's score data, σ Y represents the standard deviation of the doctor's Y score data, ρ X,Y represents the correlation between doctor X's score data and doctor Y's score data;

[0039] A correlation threshold between the doctor scoring data is set, and the fused multimodal medical images below the set threshold are removed.

[0040] Preferably, the step of extracting the contour of the fused multimodal medical image to obtain the contour of the processed combined medical image comprises the following steps:

[0041] S321, performing image binarization processing on the fused multimodal medical image;

[0042] An initial grayscale threshold k is selected to classify all pixels in the multimodal medical image into two categories C1 and C2;

[0043] Set C1 to be the pixel category that is less than or equal to the grayscale threshold k, and C2 to be the pixel category that is greater than the grayscale threshold k;

[0044] Set the grayscale mean of pixel category C1 to h1, the grayscale mean of pixel category C2 to h2, and the global grayscale mean to h3;

[0045] The probability that a pixel in the fused multimodal medical image belongs to pixel category C1 is set to p1, and the probability that a pixel belongs to pixel category C2 is set to p2;

[0046] The binarization formula is as follows:

[0047] h3=h1×p1+h2×p2;

[0048]

[0049] Wherein, η represents the binarization threshold;

[0050] The grayscale values ​​greater than the binarization threshold are set to 255, and the grayscale values ​​less than or equal to the binarization threshold are set to 0;

[0051] Summarizing the pixels in the binarized and fused multimodal medical image to obtain a binarized and fused multimodal medical image;

[0052] S322, performing contour extraction on the fused multimodal medical image after the binarization processing;

[0053] Create a one-dimensional array to record the grayscale values ​​of the eight neighborhoods around each pixel in the binarized multimodal medical image. If the grayscale values ​​of the eight neighborhoods around a pixel are the same as the grayscale value of the center point, the pixel is considered to be inside the object and deleted.

[0054] When the grayscale values ​​of the eight neighborhoods around a pixel are different from the grayscale value of the center point, the pixel is considered to be at the edge of the object and is retained.

[0055] Each pixel point in the binarized and fused multimodal medical image is traversed and summarized to obtain the contour of the processed combined medical image.

[0056] The present invention calculates the objective score of each fused multimodal medical image by processing the doctor scoring data corresponding to each fused multimodal medical image, and processes the fused multimodal medical image. At the same time, for the processed fused multimodal medical image, the contour of the processed fused multimodal medical image is extracted by contour extraction, thereby improving the accuracy of interventional treatment guidance.

[0057] Preferably, comparing the contour of the multimodal medical image of the patient's affected part acquired in real time with the contour of the processed combined medical image to determine the type of the multimodal medical image of the patient's affected part acquired in real time comprises the following steps:

[0058] S51, matching the contour of the multimodal medical image of the patient's affected part acquired in real time with the contour of the processed combined medical image;

[0059] S52: Determine the type of the multimodal medical image of the patient's affected part acquired in real time according to the degree of overlap of the contour lines.

[0060] Preferably, determining the type of the multimodal medical image of the patient's affected part acquired in real time according to the degree of overlap of the contour lines comprises the following steps:

[0061] Set the contour of the processed combined medical image to be L m , the multimodal medical image of the patient's affected part collected in real time is D, and the contour of the multimodal medical image of the patient's affected part collected in real time is L D ; Use the similarity function S(Θ) to judge the contour image L m , L D degree of overlap;

[0062]

[0063] Where S(Θ) represents the contour image L m , L D Similarity, A0(Θ) is the similarity between the two contours L m With L D The area of ​​the overlapping part, A(L m ) and A(L D ) are the areas of the corresponding contours; 0≤S(Θ)≤1, the larger S(Θ), the larger L m and L D The more similar, the more m and L D When they coincide, S(Θ)=1;

[0064] Set the similarity threshold, when L m and L D When the similarity is greater than the set threshold, it means L m and LD The type is consistent.

[0065] The present invention matches the contours of the multimodal medical images of the patient's affected part collected in real time with the contours of the processed medical images, calculates the similarity between the two sets of contour images by contour comparison, and judges the type of the multimodal medical images of the patient's affected part collected in real time based on the calculated similarity, thereby ensuring the accuracy of interventional treatment guidance.

[0066] Preferably, after determining the type of multimodal medical image acquired in real time, establishing precise positioning between the image and the patient, and guiding treatment based on the established positioning includes the following steps:

[0067] S61. Measure the position and posture of the patient and establish a three-dimensional coordinate system of the patient's position and posture;

[0068] S62. Based on the established three-dimensional coordinate system of the patient's position and posture, the contour of the multimodal medical image of the patient's affected part acquired in real time after the type is determined is projected into the established three-dimensional coordinate system, and the relationship between the projection and the three-dimensional coordinate system is calculated.

[0069] S63. Based on the relationship between the calculated projection and the three-dimensional coordinate system, real-time acquisition, real-time projection, and real-time positioning guidance of the affected part are performed.

[0070] Preferably, calculating the relationship between the projection and the three-dimensional coordinate system comprises the following steps:

[0071] The relationship between the calculated projection and the three-dimensional coordinate system is shown below;

[0072] Let Q be the coordinate of the affected part in the three-dimensional coordinate system, and q be the coordinate of the contour of the multimodal medical image of the affected part of the patient acquired in real time in the three-dimensional coordinate system;

[0073] Q=(x Q ,y Q ,z Q ,w Q );

[0074] q=(x q ,y q ,w q );

[0075] Among them, x Q ,y Q ,z Q Respectively represent the horizontal, vertical and vertical coordinates of the affected area in the three-dimensional coordinate system; q ,y q They represent the horizontal and vertical coordinates of the contour of the multimodal medical image of the patient's affected part collected in real time in the three-dimensional coordinate system, w QIndicates the wheelbase parameter of the established three-dimensional coordinate system, w q represents the axis distance parameter of the multimodal medical image of the patient's affected part acquired in real time;

[0076] Set when w q =w Q When the multimodal medical image of the patient's affected part is collected in real time, the contour projection is coaxial with the established three-dimensional coordinate system;

[0077] When the contour projection of the multimodal medical image of the patient's affected area acquired in real time is coaxial with the established three-dimensional coordinate system, the relationship between the two sets of coordinates is as follows:

[0078]

[0079] Among them, f x represents the x-axis camera focal length, f y represents the y-axis camera focal length, e x Indicates the horizontal coordinate of the center point of the affected area on the x-axis, e y Indicates the vertical coordinate of the center point of the affected area on the y-axis; Represents the rotation vector matrix, r 11 Represents the rotation vector of the first row and first column in the rotation vector matrix, Represents the translation vector matrix, and t1 represents the translation vector of the first row.

[0080] The present invention measures the patient's position and posture to establish a three-dimensional coordinate system for the patient's position and posture, and projects the contours of multimodal medical images collected in real time to calculate the relationship between the projection and the three-dimensional coordinate system, thereby completing the conversion between the projection and the three-dimensional coordinate system and achieving precise positioning of the affected area, thereby improving the real-time and accuracy of interventional treatment guidance.

[0081] The present invention discloses an image positioning guidance system based on multimodal image fusion, comprising: a fusion scoring module, a data processing module, a contour extraction module, a type determination module and a precise positioning module;

[0082] The fusion scoring module is used to fuse multimodal medical images, obtain fused multimodal medical images and perform doctor scoring;

[0083] The data processing module is used to process the collected multimodal medical images and the doctor scoring data of the multimodal medical images;

[0084] The contour extraction module is used to extract contours from the processed multimodal medical image;

[0085] The type determination module is used to determine the type of the multimodal medical image acquired in real time by contour comparison;

[0086] The precise positioning module is used to precisely locate the affected area through three-dimensional coordinate conversion and real-time acquisition and projection.

[0087] (3) Beneficial effects

[0088] Compared with the existing technology, it has the following beneficial effects:

[0089] 1. The invention fuses multimodal medical images to obtain a fused multimodal medical image and performs doctor scoring, and constructs a multimodal medical image sample set. At the same time, the fused multimodal medical image and the corresponding doctor scoring data in the constructed multimodal medical image sample set are processed to obtain the contour of the processed combined medical image. At the same time, the contour of the multimodal medical image of the patient's affected part collected in real time is extracted in the same way, and the contour of the multimodal medical image of the patient's affected part extracted in real time is compared with the contour of the extracted and processed combined medical image to determine the type of the multimodal medical image of the patient's affected part collected in real time. After determining the type of the multimodal medical image of the patient's affected part collected in real time, precise positioning is established between the image and the patient, and treatment guidance is performed based on the established positioning, thereby improving the real-time and accuracy of interventional treatment guidance.

[0090] 2. The invention calculates the objective score of each fused multimodal medical image by processing the doctor's scoring data corresponding to each fused multimodal medical image, and processes the fused multimodal medical image. At the same time, for the processed fused multimodal medical image, the contour of the processed fused multimodal medical image is extracted by contour extraction, thereby improving the accuracy of interventional treatment guidance.

[0091] 3. The invention matches the contours of the multimodal medical images of the patient's affected part collected in real time with the contours of the processed medical images, calculates the similarity between the two sets of contour images by contour comparison, and determines the type of the multimodal medical images of the patient's affected part collected in real time based on the calculated similarity, thereby ensuring the accuracy of interventional treatment guidance.

[0092] 4. This invention establishes a three-dimensional coordinate system of the patient's position and posture by measuring the patient's position and posture, and calculates the relationship between the projection and the three-dimensional coordinate system by projecting the contours of the multimodal medical image collected in real time, completing the conversion between the projection and the three-dimensional coordinate system, and realizing precise positioning of the affected area, thereby improving the real-time and accuracy of interventional treatment guidance. BRIEF DESCRIPTION OF THE DRAWINGS

[0093] Figure 1 The figure is a flow chart of the structure of an image positioning and guidance system based on multimodal image fusion according to the present invention. DETAILED DESCRIPTION

[0094] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0095] Example 1

[0096] This embodiment discloses an image processing method based on multimodal image fusion, which specifically includes the following steps:

[0097] S1. Establish a multimodal medical image set and select the image types that need to be fused to perform image fusion to obtain a fused multimodal medical image;

[0098] Selecting the image type to be fused and performing image fusion includes the following steps:

[0099] Image fusion based on pixel weighted average method;

[0100] Set the weight of each image type;

[0101] Select image types with the same image size for fusion;

[0102] The fused image formula is as follows:

[0103] F(α,β)=ω1B(α,β)+ω2E(α,β);

[0104] Among them, F(α, β) represents the pixel at (α, β) in the fused image, B(α, β) represents the pixel at (α, β) in image B, E(α, β) represents the pixel at (α, β) in image E, and ω1 and ω2 represent the weights of the corresponding image types.

[0105] S2. Perform doctor ratings on the fused multimodal medical images, collect doctor rating data, and construct a multimodal medical image sample set;

[0106] Set the value range of the collected doctor scoring data corresponding to the multimodal medical images to [1, 5];

[0107] Setting "5 points" means that the image quality is excellent and does not affect the judgment;

[0108] Setting "4 points" means that the image quality is good and does not affect the judgment;

[0109] Setting "3 points" indicates that the image quality is poor and affects the judgment;

[0110] Setting "2 points" indicates poor image quality, which significantly affects judgment;

[0111] Setting "1 point" means that the image quality is extremely poor and cannot be judged;

[0112] S3, processing the multimodal medical image and the corresponding doctor scoring data after fusion in the constructed multimodal medical image sample set to obtain the outline of the processed combined medical image;

[0113] Processing the multimodal medical images and the doctor rating data corresponding to the multimodal medical images in constructing the multimodal medical image sample set includes the following steps:

[0114] S31, processing the doctor scoring data corresponding to the fused multimodal medical image;

[0115] S311, summarizing the doctor rating data corresponding to each fused multimodal medical image, and calculating the subjective score of each fused multimodal medical image;

[0116] The formula for calculating the subjective score is as follows:

[0117]

[0118] in, represents the calculated subjective score of the fused multimodal medical image, u i represents the scoring data of the i-th doctor, and T represents the number of doctors;

[0119] Setting a subjective score threshold of the fused multimodal medical image, and removing the fused multimodal medical image with a subjective score lower than the set subjective score threshold;

[0120] S312. Calculating an objective evaluation score for each fused multimodal medical image based on the subjective score of each fused multimodal medical image;

[0121] The standard deviation of the subjective scores is calculated based on the subjective scores of each fused multimodal medical image. The calculation formula is as follows:

[0122]

[0123] Where σ represents the standard deviation of the calculated subjective scores;

[0124] The correlation of doctor rating data was calculated based on the Pearson correlation coefficient. The calculation formula is as follows:

[0125]

[0126] Among them, cov(X,Y) represents the covariance between the doctor X score data and the doctor Y score data, σ X represents the standard deviation of doctor X's score data, σ Y represents the standard deviation of the doctor's Y score data, ρ X,Y represents the correlation between doctor X's score data and doctor Y's score data;

[0127] Setting a correlation threshold between doctor-scoring data and removing fused multimodal medical images with values ​​below the set threshold;

[0128] S32, performing contour extraction on the fused multimodal medical image to obtain the contour of the processed combined medical image;

[0129] S321, performing image binarization processing on the fused multimodal medical image;

[0130] An initial grayscale threshold k is selected to classify all pixels in the multimodal medical image into two categories C1 and C2;

[0131] Set C1 to be the pixel category that is less than or equal to the grayscale threshold k, and C2 to be the pixel category that is greater than the grayscale threshold k;

[0132] Set the grayscale mean of pixel category C1 to h1, the grayscale mean of pixel category C2 to h2, and the global grayscale mean to h3;

[0133] The probability that a pixel in the fused multimodal medical image belongs to pixel category C1 is set to p1, and the probability that a pixel belongs to pixel category C2 is set to p2;

[0134] The binarization formula is as follows:

[0135] h3=h1×p1+h2×p2;

[0136]

[0137] Wherein, η represents the binarization threshold;

[0138] The grayscale values ​​greater than the binarization threshold are set to 255, and the grayscale values ​​less than or equal to the binarization threshold are set to 0;

[0139] Summarizing the pixels in the binarized and fused multimodal medical image to obtain a binarized and fused multimodal medical image;

[0140] S322, performing contour extraction on the fused multimodal medical image after the binarization processing;

[0141] Create a one-dimensional array to record the grayscale values ​​of the eight neighborhoods around each pixel in the binarized multimodal medical image. If the grayscale values ​​of the eight neighborhoods around a pixel are the same as the grayscale value of the center point, the pixel is considered to be inside the object and deleted.

[0142] When the grayscale values ​​of the eight neighborhoods around a pixel are different from the grayscale value of the center point, the pixel is considered to be at the edge of the object and is retained.

[0143] Traversing and summarizing each pixel point in the fused multimodal medical image after binarization processing to obtain the contour of the processed combined medical image;

[0144] S4. Using an image acquisition device, collect and process multimodal medical images of the patient's affected area in real time to obtain a contour of the multimodal medical image of the patient's affected area collected in real time;

[0145] S5. comparing the contour of the multimodal medical image of the patient's affected part acquired in real time with the contour of the processed combined medical image to determine the type of the multimodal medical image of the patient's affected part acquired in real time;

[0146] Comparing the contour of the multimodal medical image of the affected part of the patient acquired in real time with the contour of the processed combined medical image to determine the type of the multimodal medical image of the affected part of the patient acquired in real time includes the following steps:

[0147] S51, matching the contour of the multimodal medical image of the patient's affected part extracted in real time with the contour of the processed medical image;

[0148] S52, determining the type of the multimodal medical image of the patient's affected part acquired in real time according to the degree of overlap of the contour lines;

[0149] Set the contour of the processed combined medical image to be L m , the multimodal medical image of the patient's affected part collected in real time is D, and the contour of the multimodal medical image of the patient's affected part collected in real time is L D ; Use the similarity function S(Θ) to judge the contour image L m , L D degree of overlap;

[0150]

[0151] Where S(Θ) represents the contour image L m , L D Similarity, A0(Θ) is the similarity between the two contours L m With L D The area of ​​the overlapping part, A(L m ) and A(L D) are the areas of the corresponding contours; 0≤S(Θ)≤1, the larger S(Θ), the larger L m and L D The more similar, the more m and L D When they coincide, S(Θ)=1;

[0152] Set the similarity threshold, when L m and L D When the similarity is greater than the set threshold, it means L m and L D The type is consistent.

[0153] Furthermore, in some embodiments, the method further includes S6: after determining the type of the multimodal medical image acquired in real time, establishing a precise positioning between the image and the patient, and guiding treatment based on the established positioning;

[0154] After determining the type of multimodal medical image to be acquired in real time, establishing precise positioning between the image and the patient, and guiding treatment based on the established positioning include the following steps:

[0155] S61. Measure the position and posture of the patient and establish a three-dimensional coordinate system of the patient's position and posture;

[0156] S62: Based on the established three-dimensional coordinate system of the patient's position and posture, project the contour of the multimodal medical image acquired in real time after the type is determined into the established three-dimensional coordinate system, and calculate the relationship between the projection and the three-dimensional coordinate system;

[0157] The relationship between the calculated projection and the three-dimensional coordinate system is shown below;

[0158] Let Q be the coordinate of the affected part in the three-dimensional coordinate system, and q be the coordinate of the contour of the multimodal medical image of the affected part of the patient acquired in real time in the three-dimensional coordinate system;

[0159] Q=(x Q ,y Q ,z Q ,w Q );

[0160] q=(x q ,y q ,w q );

[0161] Among them, x Q ,y Q ,z Q Respectively represent the horizontal, vertical and vertical coordinates of the affected area in the three-dimensional coordinate system; q ,y q They represent the horizontal and vertical coordinates of the contour of the multimodal medical image of the patient's affected part collected in real time in the three-dimensional coordinate system, w QIndicates the wheelbase parameter of the established three-dimensional coordinate system, w q represents the axis distance parameter of the multimodal medical image of the patient's affected part acquired in real time;

[0162] Set when w q =w Q When the multimodal medical image of the patient's affected part is collected in real time, the contour projection is coaxial with the established three-dimensional coordinate system;

[0163] When the contour projection of the multimodal medical image of the patient's affected area acquired in real time is coaxial with the established three-dimensional coordinate system, the relationship between the two sets of coordinates is as follows:

[0164]

[0165] Among them, f x represents the x-axis camera focal length, f y represents the y-axis camera focal length, e x Indicates the horizontal coordinate of the center point of the affected area on the x-axis, e y Indicates the vertical coordinate of the center point of the affected area on the y-axis; Represents the rotation vector matrix, r 11 Represents the rotation vector of the first row and first column in the rotation vector matrix, Represents the translation vector matrix, t1 represents the translation vector of the first row;

[0166] S63. Based on the relationship between the calculated projection and the three-dimensional coordinate system, real-time acquisition and real-time projection of the affected area are performed to provide real-time guidance for treatment of the affected area.

[0167] Example 2

[0168] This embodiment discloses an image positioning guidance system based on multimodal image fusion, comprising: a fusion scoring module, a data processing module, a contour extraction module, a type determination module, and a precise positioning module;

[0169] The fusion scoring module is used to fuse multimodal medical images, obtain fused multimodal medical images and perform doctor scoring;

[0170] The data processing module is used to process the collected multimodal medical images and the doctor scoring data of the multimodal medical images;

[0171] The contour extraction module is used to extract contours from the processed multimodal medical image;

[0172] The type determination module is used to determine the type of the multimodal medical image acquired in real time by contour comparison;

[0173] The precise positioning module is used to precisely locate the affected area through three-dimensional coordinate conversion and real-time acquisition and projection.

[0174] While the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and alterations may be made to the embodiments without departing from the principles and spirit of the invention, and the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. An image processing method based on multimodal image fusion, characterized in that: The following steps are involved: S1. Establish a multimodal medical image set and select the image types that need to be fused to perform image fusion to obtain a fused multimodal medical image; S2. Perform doctor ratings on the fused multimodal medical images, collect doctor rating data, and construct a multimodal medical image sample set; S3, processing the multimodal medical image and the corresponding doctor scoring data after fusion in the constructed multimodal medical image sample set to obtain the outline of the processed combined medical image; Calculating the objective score of the multimodal medical image and judging it according to the set objective score threshold, when the objective score is lower than the objective score threshold, deleting the multimodal medical image, and when the objective score is not lower than the objective score threshold, extracting the contour of the multimodal medical image; S4. Using an image acquisition device, collect and process multimodal medical images of the patient's affected area in real time to obtain a contour of the multimodal medical image of the patient's affected area collected in real time; S5. comparing the contour of the multimodal medical image of the patient's affected part acquired in real time with the contour of the processed combined medical image to determine the type of the multimodal medical image of the patient's affected part acquired in real time; S6. After determining the type of the multimodal medical image of the patient's affected area acquired in real time, establishing precise positioning between the image and the patient, including the following steps: S61. Measure the position and posture of the patient and establish a three-dimensional coordinate system of the patient's position and posture; S62: Based on the established three-dimensional coordinate system of the patient's position and posture, project the contour of the multimodal medical image of the patient's affected area acquired in real time after the type is determined into the established three-dimensional coordinate system, and calculate the relationship between the projection and the three-dimensional coordinate system; S63, based on the relationship between the calculated projection and the three-dimensional coordinate system, performing real-time acquisition, real-time projection, and real-time positioning guidance of the affected part; The calculation of the relationship between the projection and the three-dimensional coordinate system comprises the following steps: set up The coordinates of the affected area in the three-dimensional coordinate system, The coordinates of the contour of the multimodal medical image of the patient's affected part collected in real time in the three-dimensional coordinate system; ； ； in, Respectively represent the horizontal, vertical and vertical coordinates of the affected area in the three-dimensional coordinate system; They represent the horizontal and vertical coordinates of the contour of the multimodal medical image of the patient's affected part collected in real time in the three-dimensional coordinate system, Indicates the wheelbase parameter of the established three-dimensional coordinate system. represents the axis distance parameter of the multimodal medical image of the patient's affected part acquired in real time; Set when = When the multimodal medical image of the patient's affected part is collected in real time, the contour projection is coaxial with the established three-dimensional coordinate system; When the contour projection of the multimodal medical image of the patient's affected area acquired in real time is coaxial with the established three-dimensional coordinate system, the relationship between the two sets of coordinates is as follows: ； in, express Axis camera focal length, express Axis camera focal length, express The horizontal coordinate of the center point of the affected area on the axis, express The vertical coordinate of the center point of the affected area on the axis; represents the rotation vector matrix, Represents the rotation vector of the first row and first column in the rotation vector matrix, represents the translation vector matrix, Represents the translation vector of the first row.

2. The image processing method based on multimodal image fusion according to claim 1, characterized in that: The selecting of the image types to be fused for image fusion comprises the following steps: Image fusion based on pixel weighted average method; Set the weight of each image type; Select image types with the same image size for fusion; The fused image formula is as follows: ; in, Represents the fused image Pixels at Representing an image middle Pixels at Representing an image middle Pixels at , They respectively represent the weights of the corresponding image types.

3. The image processing method based on multimodal image fusion according to claim 1, characterized in that: The processing of the fused multimodal medical images and the corresponding doctor scoring data in the constructed multimodal medical image sample set includes the following steps: S31, processing the doctor scoring data corresponding to the fused multimodal medical image; S32 . Perform contour extraction on the fused multimodal medical image to obtain the contour of the processed combined medical image.

4. The image processing method based on multimodal image fusion according to claim 3, characterized in that: The processing of the doctor scoring data corresponding to the fused multimodal medical image comprises the following steps: S311, summarizing the doctor rating data corresponding to each fused multimodal medical image, and calculating the subjective score of each fused multimodal medical image; The formula for calculating the subjective score is as follows: ； in, represents the calculated subjective score of the fused multimodal medical image, Indicates the Doctors' rating data, represents the number of doctors; Setting a subjective score threshold of the fused multimodal medical image, and removing the fused multimodal medical image with a subjective score lower than the set subjective score threshold; S312. Calculating an objective evaluation score for each fused multimodal medical image based on the subjective score of each fused multimodal medical image; The standard deviation of the subjective scores is calculated based on the subjective scores of each fused multimodal medical image. The calculation formula is as follows: ； in, represents the standard deviation of the calculated subjective scores; The correlation of doctor rating data was calculated based on the Pearson correlation coefficient. The calculation formula is as follows: ； in, Indicates doctor Scoring data and doctors The covariance between the rating data, Indicates doctor The standard deviation of the rating data, Indicates doctor The standard deviation of the rating data, Indicates doctor Scoring data and doctors Correlation between scoring data; A correlation threshold between the doctor scoring data is set, and the fused multimodal medical images below the set threshold are removed.

5. The image processing method based on multimodal image fusion according to claim 3, characterized in that: The step of extracting the contour of the fused multimodal medical image to obtain the contour of the processed combined medical image comprises the following steps: S321, performing image binarization processing on the fused multimodal medical image; Select the initial grayscale threshold Classify all pixels in multimodal medical images into two categories and ; set up is less than or equal to the grayscale threshold Pixel categories, is greater than the grayscale threshold Pixel category; Set pixel category The grayscale mean is , pixel category The grayscale mean is , the global grayscale mean is ; Assume that the pixels in the fused multimodal medical image belong to the pixel category The probability of , belongs to the pixel category The probability of ; The binarization formula is as follows: ； ； in, represents the binarization threshold; The grayscale values ​​greater than the binarization threshold are set to 255, and the grayscale values ​​less than or equal to the binarization threshold are set to 0; Summarizing the pixels in the binarized and fused multimodal medical image to obtain a binarized and fused multimodal medical image; S322, performing contour extraction on the binarized fused multimodal medical image to obtain the contour of the processed combined medical image; Create a one-dimensional array to record the grayscale values ​​of the eight neighborhoods around each pixel of the binarized and fused multimodal medical image. If the grayscale values ​​of the eight neighborhoods around a pixel are the same as the grayscale value of the center point, the pixel is considered to be inside the object and deleted. When the grayscale values ​​of the eight neighborhoods around a pixel are different from the grayscale value of the center point, the pixel is considered to be at the edge of the object and is retained. Each pixel point in the binarized and fused multimodal medical image is traversed and summarized to obtain the contour of the processed combined medical image.

6. The image processing method based on multimodal image fusion according to claim 1, characterized in that: The step of comparing the contour of the multimodal medical image of the affected part of the patient acquired in real time with the contour of the processed combined medical image to determine the type of the multimodal medical image of the affected part of the patient acquired in real time comprises the following steps: S51, matching the contour of the multimodal medical image of the patient's affected part acquired in real time with the contour of the processed combined medical image; S52: Determine the type of the multimodal medical image of the patient's affected part acquired in real time according to the degree of overlap of the contour lines.

7. The image processing method based on multimodal image fusion according to claim 6, characterized in that: Determining the type of the multimodal medical image of the patient's affected part acquired in real time according to the degree of overlap of the contour lines comprises the following steps: The contour of the processed combined medical image is set to , the multimodal medical images of the patient's affected area collected in real time are , the contour of the multimodal medical image of the patient's affected area collected in real time is ; Through the similarity function To judge the contour image 、 degree of overlap; ； in, Represents a contour image 、 The similarity of For two contours and The area of ​​the overlapping part, and are the areas of the corresponding contours respectively; , The bigger, and The more similar, the and When overlapped, ; Set the similarity threshold. and When the similarity is greater than the set threshold, it means and The type is consistent.

8. An image positioning and guidance system implementing the image processing method based on multimodal image fusion according to any one of claims 1 to 7, characterized in that: include: Fusion scoring module, data processing module, contour extraction module, type determination module and precise positioning module; The fusion scoring module is used to fuse multimodal medical images, obtain fused multimodal medical images and perform doctor scoring; The data processing module is used to process the fused multimodal medical images and the corresponding doctor scoring data; The contour extraction module is used to extract contours from the processed multimodal medical image; The type determination module is used to determine the type of the multimodal medical image of the patient's affected part acquired in real time by contour comparison; The precise positioning module is used to precisely locate the affected area through three-dimensional coordinate conversion and real-time acquisition and projection.

Images