A multispectral multi-channel imaging method and system based on image fusion

Abstract

The present invention discloses a multispectral multi-channel imaging method based on image fusion, which includes the following steps: S1, using a spectral camera to collect multi-band multispectral images and hyperspectral images of an object to be measured; S2, performing effective feature extraction and screening on the multispectral images to obtain an effective spectral image; S3, using the HIS algorithm to fuse the effective spectral images to obtain a high-frequency multispectral image; S4, fusing the high-frequency multispectral image with the hyperspectral image to obtain a fused image; The present invention also discloses a multispectral multi-channel imaging system based on image fusion. By acquiring multispectral images of different bands and performing effective feature extraction and screening, the present invention can better reflect the spectral information required for medical multispectral images, thereby ensuring imaging quality and effectively improving imaging efficiency.

Description

Technical Field

[0001] The present invention relates to the technical field of medical spectral imaging, and in particular to a multi-spectral multi-channel imaging method and system based on image fusion. Background Art

[0002] Multispectral image fusion is a type of digital image fusion technology. It is used to enhance multispectral images. Images in different wavelengths only focus on a portion of the information about the object being photographed, often resulting in unsatisfactory information presentation. Multispectral image fusion allows the image information of the object to be clearly and accurately presented in a single image, effectively reducing interference from adverse factors such as image background and periodic noise. This allows the desired information to be more prominent, thereby improving people's ability to discern information. The biggest difference between this technology and conventional imaging techniques is that multispectral imaging can obtain a high-resolution spectrum for each pixel in each image, rather than the red, blue, and green images seen by the naked eye.

[0003] With the development of multispectral imaging technology, its applications in various fields are gradually increasing. Multispectral imaging technology can be applied to plant virus monitoring, biomedicine, and other fields. However, the application of multispectral imaging technology in various fields still faces many challenges, such as the continuous improvement of spectral and image resolution, the huge amount of data, and the calibration of spectral imagers, which are difficult to overcome in both storage and computing processes.

[0004] Most existing medical imaging systems use standard color visible light imaging, which can only produce standard color images and lack the ability to detect details outside the visible spectrum. The emergence of multispectral imaging technology has effectively addressed this issue. Based on multispectral illumination, it illuminates different diseased tissues and combines the characteristics of corresponding fluorescent reagents to obtain images of specific lesions that are not clearly visible under visible light. These images are then processed to enhance the boundaries between lesion features and non-lesion areas.

[0005] Existing multispectral imaging systems cannot simultaneously image multiple spectra, requiring constant mode switching when examining internal tissues, which is cumbersome. Multispectral mode can only display one specific light at a time and cannot simultaneously image multiple spectra. The resulting image quality is poor, making it impossible to effectively observe the edge contours of each tissue in the image, resulting in an inability to detect the location of lesions.

[0006] Currently, no effective solutions have been proposed for the problems in related technologies. Summary of the Invention

[0007] In response to the problems in the related art, the present invention proposes a multi-spectral multi-channel imaging method and system based on image fusion to overcome the above technical problems existing in the existing related art.

[0008] To this end, the specific technical solutions adopted in the present invention are as follows:

[0009] According to one aspect of the present invention, a multispectral multi-channel imaging method based on image fusion is provided, the method comprising the following steps:

[0010] S1, using a spectral camera to collect multi-band multispectral images and hyperspectral images of the object to be measured;

[0011] S2. Extracting and screening effective features of the multispectral image to obtain an effective spectral image;

[0012] S3, using the HIS algorithm to fuse the effective spectrum images to obtain a high-frequency multispectral image;

[0013] S4. Fusing the high-frequency multispectral image with the hyperspectral image to obtain a fused image.

[0014] Furthermore, extracting and screening effective features from the multispectral image to obtain an effective spectral image comprises the following steps:

[0015] S21, inputting the multispectral images of several different bands into a feature extraction model;

[0016] S22, calculating the band index of each of the multispectral images using the feature extraction model;

[0017] S23 , sorting the band indices in descending order, and selecting the top five corresponding multispectral images as the final selected valid spectral images.

[0018] Furthermore, calculating the band index of each multispectral image using the feature extraction model includes the following steps:

[0019] S221, grouping a number of multispectral images of different bands in order from small to large into m groups, with the number of bands in each group being n1, n2, ..., n m ;

[0020] S222. Calculate the average value of the sum of the absolute values ​​of the correlation coefficients between the i-th band and the other bands in the group. The calculation formula is:

[0021]

[0022] S223. Calculate the correlation coefficients between different bands in each group and other bands, and use them as the denominator of the band index. The calculation formula is:

[0023] R i =R w +R a

[0024] S224. Calculate the band index using the following formula:

[0025]

[0026] Where R w It represents the average value of the sum of the absolute values ​​of the correlation coefficients between the i-th band and other bands in the group;

[0027] R a It represents the sum of the absolute values ​​of the correlation coefficients between the i-th band and other bands in the group;

[0028] R w Denominator of the band index;

[0029] P i Indicates the band index;

[0030] n represents the number of multispectral images in each group;

[0031] m represents the number of groups;

[0032] ρ ij represents the correlation coefficient between channels i and j;

[0033] σ i represents the mean square error of the i-th band;

[0034] Both i and j represent the bands of the multispectral image.

[0035] Furthermore, the method of fusing the effective spectral images using the HIS algorithm to obtain a high-frequency multispectral image includes the following steps:

[0036] S31, performing spatial registration on the effective spectrum image;

[0037] S32, performing HIS transformation on the effective spectrum image to obtain three component images: H, I, and S;

[0038] S33, performing histogram matching on the I components of the plurality of effective spectrum images;

[0039] S34, replacing the I components of the other effective spectrum images with the I component of the basic spectrum image;

[0040] S35 , performing a transformation on the H and S components of the basic spectrum image and the other effective spectrum images to generate a fused high-frequency multispectral image.

[0041] Furthermore, the basic spectrum image is the effective spectrum image with the highest band coefficient among the five effective spectrum images.

[0042] Furthermore, fusing the high-frequency multispectral image with the hyperspectral image to obtain a fused image includes the following steps:

[0043] S41, preprocessing the high-frequency multispectral image and the hyperspectral image respectively;

[0044] S42, converting the high-frequency multispectral image into a synthetic spectral image through linear approximation;

[0045] S43: Fusing the synthetic spectral image with the hyperspectral image to obtain a fused image.

[0046] Furthermore, preprocessing the high-frequency multispectral image and the hyperspectral image separately includes the following steps:

[0047] S411, performing geometric correction on the high-frequency multispectral image and the hyperspectral image;

[0048] S412: Perform image registration on the high-frequency multispectral image and the hyperspectral image.

[0049] Furthermore, converting the high-frequency multispectral image into a synthetic spectral image through linear approximation includes the following steps:

[0050] S421, confirming the spectrum matrix of the high-frequency multispectral image;

[0051] S422, obtaining an estimation matrix of multispectral approximate conversion to hyperspectral by performing an inverse transformation on the spectrum matrix;

[0052] S423, multiplying the estimation matrix by the spectrum of the high-frequency multispectral image to obtain a spectrum estimation vector of the hyperspectral image;

[0053] S424 , calculating spectrum estimation vectors one by one according to the spectrum of the high-frequency multi-spectral image, and finally obtaining a synthetic spectrum image.

[0054] Furthermore, fusing the synthetic spectral image with the hyperspectral image to obtain a fused image comprises the following steps:

[0055] S431, using a high-pass filter to select a high-frequency portion of the hyperspectral image;

[0056] S432, selecting a low-frequency portion of the synthetic spectrum image using a low-pass filter;

[0057] S433: Fusing and adding the high-frequency part and the low-frequency part to obtain a final fused image.

[0058] According to another aspect of the present invention, a multispectral multi-channel imaging system based on image fusion is also provided, which comprises the following components: a multispectral camera, a hyperspectral camera, a multispectral extraction module, a multispectral fusion module and a comprehensive image fusion module;

[0059] The multispectral camera is used to capture multispectral images of different wavelengths.

[0060] The hyperspectral camera is used to capture and collect hyperspectral images;

[0061] The multispectral extraction module is used to extract and screen the features of the multispectral image;

[0062] The multispectral fusion module is used to synthesize high-frequency multispectral images;

[0063] The comprehensive image fusion module is used to fuse the high-frequency multispectral image with the hyperspectral image to obtain a fused image.

[0064] The beneficial effects of the present invention are as follows: by acquiring multispectral images of different bands and performing effective feature extraction and screening, the spectral information required for medical multispectral images can be better reflected, thereby ensuring imaging quality and effectively improving imaging efficiency; at the same time, the effective spectral images after screening are initially fused, which can provide complete and clear images more concisely, effectively improve the resolution of multispectral images, and enhance the detail features of multispectral images; finally, through fusion with hyperspectral images, a fused image with high resolution and high spectral details is achieved, that is, a clearer and more complete medical spectral image is obtained, thereby making medical images have stronger high speed and high definition. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0066] Figure 1 is a flow chart of a multispectral multi-channel imaging method based on image fusion according to an embodiment of the present invention;

[0067] Figure 2This is a system block diagram of a multi-spectral multi-channel imaging system based on image fusion according to an embodiment of the present invention.

[0068] In the picture:

[0069] 1. Multispectral camera; 2. Hyperspectral camera; 3. Multispectral extraction module; 4. Multispectral fusion module; 5. Comprehensive image fusion module. DETAILED DESCRIPTION

[0070] To further illustrate each embodiment, the present invention provides drawings, which are part of the disclosure of the present invention. They are mainly used to illustrate the embodiments and can be used in conjunction with the relevant descriptions in the specification to explain the operating principles of the embodiments. By referring to these contents, ordinary technicians in this field should be able to understand other possible implementation methods and advantages of the present invention. The components in the figures are not drawn to scale, and similar component symbols are generally used to represent similar components.

[0071] According to one embodiment of the present invention, a multi-spectral multi-channel imaging method based on image fusion is provided. The present invention is further described with reference to the accompanying drawings and specific embodiments. Figure 1 As shown, the method includes the following steps:

[0072] S1, using a spectral camera to collect multi-band multispectral images and hyperspectral images of the object to be measured;

[0073] S2. Extracting and screening effective features of the multispectral image to obtain an effective spectral image, comprising the following steps:

[0074] S21, inputting the multispectral images of several different bands into a feature extraction model;

[0075] S22, calculating the band index of each of the multispectral images using the feature extraction model;

[0076] Because the correlations between bands within a group are strong, while the correlations between bands between groups are weak, the overall correlation of a band is primarily determined by its correlations with the bands within its group. Groups vary in size, meaning the number of bands that make up each group varies. Therefore, using the average of the sum of the absolute values ​​of the correlation coefficients between a band and the other bands within the group as the denominator of the band index more accurately reflects the overall quality of the band.

[0077] The larger the mean square error of the band index, the greater the degree of dispersion of the band and the richer the information it contains. If the absolute value of the band correlation coefficient is smaller, it means that the channel data is more independent and the information redundancy is smaller.

[0078] Calculating the band index of each multispectral image using the feature extraction model includes the following steps:

[0079] S221, grouping a number of multispectral images of different bands in order from small to large into m groups, with the number of bands in each group being n1, n2, ..., n m ;

[0080] S222. Calculate the average value of the sum of the absolute values ​​of the correlation coefficients between the i-th band and the other bands in the group. The calculation formula is:

[0081]

[0082] S223. Calculate the correlation coefficients between different bands in each group and other bands, and use them as the denominator of the band index. The calculation formula is:

[0083] R i =R w +R a

[0084] S224. Calculate the band index using the following formula:

[0085]

[0086] Where R w It represents the average value of the sum of the absolute values ​​of the correlation coefficients between the i-th band and other bands in the group;

[0087] R a It represents the sum of the absolute values ​​of the correlation coefficients between the i-th band and other bands in the group;

[0088] R w Denominator of the band index;

[0089] P i Indicates the band index;

[0090] n represents the number of multispectral images in each group;

[0091] m represents the number of groups;

[0092] ρ ij represents the correlation coefficient between channels i and j;

[0093] σ i represents the mean square error of the i-th band;

[0094] Both i and j represent the bands of the multispectral image.

[0095] S23 , sorting the band indices in descending order, and selecting the top five corresponding multispectral images as the final selected valid spectral images.

[0096] S3, using the HIS algorithm to fuse the effective spectrum images to obtain a high-frequency multispectral image;

[0097] The HIS transform, also known as the Munsell transform, is a widely used RGB color fusion transformation method. It first uses a forward transform to transform the multispectral image from the RGB three-primary color space to the HIS color space, separating the three color components of chroma (H), brightness (I), and saturation (S). The three color components separated from the multispectral image are then matched and synthesized according to a specific color fusion method. Finally, the synthesized chroma, brightness, and saturation color components are transformed back to the RCB space. The resulting fused image retains the spectral characteristics of the multispectral image while improving its resolution, thereby enhancing the detailed features of the multispectral image.

[0098] The method of fusing the effective spectral images using the HIS algorithm to obtain a high-frequency multispectral image includes the following steps:

[0099] S31, performing spatial registration on the effective spectrum image;

[0100] S32, performing HIS transformation on the effective spectrum image to obtain three component images: H, I, and S;

[0101] The H, I, and S components represent hue, brightness, and saturation, respectively. They are relatively independent. The I component primarily stores spatial detail information, while H and S primarily store color information. The main principle of HIS transform image fusion is to replace the I component of the multispectral image with a high-resolution panchromatic image.

[0102] S33, performing histogram matching on the I components of the plurality of effective spectrum images;

[0103] S34, replacing the I components of the other effective spectrum images with the I component of the basic spectrum image;

[0104] The basic spectrum image is the effective spectrum image with the highest band coefficient among the five effective spectrum images.

[0105] S35 , performing a transformation on the H and S components of the basic spectrum image and the other effective spectrum images to generate a fused high-frequency multispectral image.

[0106] S4. Fusing the high-frequency multispectral image with the hyperspectral image to obtain a fused image.

[0107] Multispectral sharpening of hyperspectral images is to fuse the spectral information of hyperspectral images with the spatial spectrum and spectral information of multispectral images.

[0108] The step of fusing the high-frequency multispectral image with the hyperspectral image to obtain a fused image comprises the following steps:

[0109] S41, preprocessing the high-frequency multispectral image and the hyperspectral image respectively, comprising the following steps:

[0110] S411, performing geometric correction on the high-frequency multispectral image and the hyperspectral image;

[0111] S412: Perform image registration on the high-frequency multispectral image and the hyperspectral image.

[0112] The general process of image registration is to project the images onto the same coordinate system after rigorous geometric correction of multi-sensor data to correct the systematic errors. Then, a small number of control points are selected on each sensor image. Accurate image registration is achieved through the automatic selection of feature points or calculation of their similarities, rough position estimation of registration points, precise determination of registration points, and estimation of registration transformation parameters.

[0113] S42, converting the high-frequency multispectral image into a synthetic spectral image through linear approximation;

[0114] The transformed synthetic hyperspectral image and the hyperspectral image have the same rank in spectrum.

[0115] The step of converting the high-frequency multispectral image into a synthetic spectral image by linear approximation comprises the following steps:

[0116] S421: Confirm the spectrum matrix of the high-frequency multispectral image. The calculation formula is:

[0117] P M =FP H +e

[0118] Where, P M The spectrum matrix representing the multispectral matrix (high-frequency multispectral image);

[0119] P H The spectrum matrix representing the hyperspectral image;

[0120] F represents the filter matrix for transforming hyperspectral into multispectral;

[0121] e represents Gaussian white noise;

[0122] S422, performing an inverse transformation on the spectrum matrix to obtain an estimation matrix for approximate conversion of multispectral to hyperspectral, the calculation formula is:

[0123]

[0124] Where G represents the estimated matrix of multispectral approximation converted to hyperspectral;

[0125] S423: Multiply the estimation matrix by the spectrum of the high-frequency multispectral image to obtain a spectrum estimation vector of the hyperspectral image. The calculation formula is:

[0126] S H =GS M

[0127] Where S H A spectrum vector representing the hyperspectral spectrum;

[0128] S M A spectrum vector representing a multispectral image;

[0129] S424 , calculating spectrum estimation vectors one by one according to the spectrum of the high-frequency multi-spectral image, and finally obtaining a synthetic spectrum image.

[0130] S43: Fusing the synthetic spectral image with the hyperspectral image to obtain a fused image.

[0131] Fusing the synthetic spectral image with the hyperspectral image to obtain a fused image comprises the following steps:

[0132] S431, using a high-pass filter to select a high-frequency portion of the hyperspectral image;

[0133] S432, selecting a low-frequency portion of the synthetic spectrum image using a low-pass filter;

[0134] S433: Fusing and adding the high-frequency part and the low-frequency part to obtain a final fused image.

[0135] According to another embodiment of the present invention, a multispectral multi-channel imaging system based on image fusion is provided, which includes the following components: a multispectral camera 1, a hyperspectral camera 2, a multispectral extraction module 3, a multispectral fusion module 4 and a comprehensive image fusion module 5;

[0136] The multispectral camera 1 is used to capture multispectral images of different wavelength bands.

[0137] The hyperspectral camera 2 is used to capture and collect hyperspectral images;

[0138] The multispectral extraction module 3 is used to extract and screen the features of the multispectral image;

[0139] The multispectral fusion module 4 is used to synthesize high-frequency multispectral images;

[0140] The comprehensive image fusion module 5 is used to fuse the high-frequency multispectral image with the hyperspectral image to obtain a fused image.

[0141] In summary, with the help of the above-mentioned technical scheme of the present invention, by acquiring multispectral images of different bands and performing effective feature extraction and screening, the spectral information required for medical multispectral images can be better reflected, thereby ensuring the imaging quality and effectively improving the imaging efficiency; at the same time, the effective spectral images after screening are initially fused, which can provide complete and clear images more concisely, effectively improve the resolution of multispectral images, and enhance the detail features of multispectral images; finally, through fusion with hyperspectral images, a fused image with high resolution and high spectral details is realized, that is, a clearer and more complete medical spectral image is obtained, so that medical images have stronger high speed and high definition.

[0142] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A multispectral multi-channel imaging method based on image fusion, characterized in that: The method comprises the following steps: S1, using a spectral camera to collect multi-band multispectral images and hyperspectral images of the object to be measured; S2. Extracting and screening effective features from the multispectral image to obtain an effective spectral image; comprising: S21. Inputting the multispectral images of several different bands into a feature extraction model; S22. Calculating the band index of each of the multispectral images using the feature extraction model; S23. Sorting the band indexes from largest to smallest, and selecting the top five corresponding multispectral images as the final selected effective spectral images; S3, using the HIS algorithm to fuse the effective spectrum images to obtain a high-frequency multispectral image; including: S31, performing spatial registration on the effective spectrum images; S32, performing HIS transformation on the effective spectrum images to obtain three component images of H, I, and S; S33, performing histogram matching on the I components of multiple effective spectrum images; S34, replacing the I components of other effective spectrum images with the I component of the basic spectrum image; S35, performing component transformation on the H and S components of the basic spectrum image and the other effective spectrum images to generate a fused high-frequency multispectral image; S4. Fusing the high-frequency multispectral image with the hyperspectral image to obtain a fused image.

2. The multispectral multi-channel imaging method based on image fusion according to claim 1, characterized in that: Calculating the band index of each multispectral image using the feature extraction model includes the following steps: S221, grouping a number of multispectral images of different bands in order from small to large into m groups, with the number of bands in each group being n1, n2, ..., n m ; S222. Calculate the average value of the sum of the absolute values ​​of the correlation coefficients between the i-th band and the other bands in the group. The calculation formula is: S223. Calculate the correlation coefficients between different bands in each group and other bands, and use them as the denominator of the band index. The calculation formula is: R i =R w +R a S224. Calculate the band index using the following formula: Where R w It represents the average value of the sum of the absolute values ​​of the correlation coefficients between the i-th band and other bands in the group; R a It represents the sum of the absolute values ​​of the correlation coefficients between the i-th band and other bands in the group; R w Denominator of the band index; P i Indicates the band index; n represents the number of multispectral images in each group; m represents the number of groups; ρ ij represents the correlation coefficient between channels i and j; σ i represents the mean square error of the i-th band; Both i and j represent the bands of the multispectral image.

3. The multi-spectral multi-channel imaging method based on image fusion according to claim 2, characterized in that: The basic spectrum image is the effective spectrum image with the highest band coefficient among the five effective spectrum images.

4. The multi-spectral multi-channel imaging method based on image fusion according to claim 3, characterized in that: Fusing the high-frequency multispectral image with the hyperspectral image to obtain a fused image comprises the following steps: S41, preprocessing the high-frequency multispectral image and the hyperspectral image respectively; S42, converting the high-frequency multispectral image into a synthetic spectral image by linear approximation; comprising: S421, confirming the spectrum matrix of the high-frequency multispectral image; S422, obtaining an estimation matrix of multispectral approximation conversion to a hyperspectral by inverse transforming the spectrum matrix; S423, multiplying the estimation matrix by the spectrum of the high-frequency multispectral image to obtain a spectrum estimation vector of the hyperspectral; S424, calculating the spectrum estimation vectors one by one according to the spectrum of the high-frequency multispectral image, and finally obtaining a synthetic spectral image; S43: Fusing the synthetic spectral image with the hyperspectral image to obtain a fused image.

5. The multi-spectral multi-channel imaging method based on image fusion according to claim 4, characterized in that: Preprocessing the high-frequency multispectral image and the hyperspectral image separately includes the following steps: S411, performing geometric correction on the high-frequency multispectral image and the hyperspectral image; S412: Perform image registration on the high-frequency multispectral image and the hyperspectral image.

6. The multi-spectral multi-channel imaging method based on image fusion according to claim 5, characterized in that: Fusing the synthetic spectral image with the hyperspectral image to obtain a fused image comprises the following steps: S431, using a high-pass filter to select a high-frequency portion of the hyperspectral image; S432, selecting a low-frequency portion of the synthetic spectrum image using a low-pass filter; S433: Fusing and adding the high-frequency part and the low-frequency part to obtain a final fused image.

7. A multispectral multi-channel imaging system based on image fusion, used for implementing the multispectral multi-channel imaging method based on image fusion according to any one of claims 1 to 6, characterized in that: The system includes the following components: a multispectral camera, a hyperspectral camera, a multispectral extraction module, a multispectral fusion module and a comprehensive image fusion module; The multispectral camera is used to capture multispectral images of different wavelengths. The hyperspectral camera is used to capture and collect hyperspectral images; The multispectral extraction module is used to extract and screen the features of the multispectral image; The multispectral fusion module is used to synthesize high-frequency multispectral images; The comprehensive image fusion module is used to fuse the high-frequency multispectral image with the hyperspectral image to obtain a fused image.

Images