Spatial frequency domain imaging method

Abstract

The invention discloses a spatial frequency domain imaging method, which comprises the following steps: acquiring first optical parameter diagrams of a to-be-detected tissue under different wavelengths, and demodulating a diffuse reflection image of the to-be-detected tissue to obtain the first optical parameter diagrams under the different wavelengths; inputting the first optical parameter diagrams under different wavelengths into a semantic segmentation network to obtain second optical parameter diagrams only containing the to-be-detected tissue under different wavelengths; taking the second optical parameter diagram under one wavelength as a reference image, and registering the second optical parameter diagrams under other wavelengths to obtain registered optical parameter diagrams with correct relative positions under different wavelengths; combining the registration optical parameter images under different wavelengths into channels to form an RGB image, and inputting the RGB image into a spectrum reconstruction network to obtain a hyperspectral image of the tissue to be detected; according to the invention, a high-quality spatial frequency domain imaging result can be quickly obtained.

Description

Technical Field

[0001] The present invention relates to the field of optical imaging technology, and in particular to a spatial frequency domain imaging method. Background Art

[0002] Spatial Frequency Domain Imaging (SFDI) is a non-contact optical imaging technology that can provide information about the composition and structure of tissues based on the spatial distribution of light in the tissue. Therefore, spatial frequency domain imaging methods show great potential in quantitative analysis of the composition and structure of human tissues, biological tissues, etc. However, converting spatial frequency domain imaging technology into clinically relevant tools is a major challenge. Since spatial frequency domain imaging methods rely on the analysis of the spatial frequency response of structured light, when measuring tissues, structured light is projected onto the surface of the tissue to obtain optical data. This requires the acquisition of multiple images with different parameters (such as wavelength, frequency, and phase). Most spatial frequency domain imaging requires at least two frequencies and three phases of structured light for projection, so that the acquired images can be demodulated to produce better results.

[0003] However, the traditional spatial frequency domain imaging method of taking multiple spatial frequency domain images requires taking tissue images at different frequencies and phases. The subject must keep the posture still when the machine is shooting. If they move, jitter errors will be generated. The delay jitter errors during multi-phase and multi-frequency shooting will affect the subsequent imaging results, and there are problems such as slow acquisition speed and motion artifacts caused by multiple imaging. With the development of technology, a single snapshot can solve the problems of jitter errors caused by phase delay and frequency delay in multiple shots for spatial frequency domain imaging. In 2013, Vervandier J et al. proposed a novel spatial frequency domain image acquisition and processing method, which can achieve one-time imaging of optical properties in the spatial frequency domain to produce results, but the imaging effect is not as good as the imaging method of collecting multiple spatial frequency domain images. In 2017, Cao Z et al. proposed a single snapshot multiple-frequency demodulation (SSMD) method, which uses the orthogonality of harmonic functions to simultaneously extract modulation transfer functions of multiple frequencies, arbitrary methods and amplitudes from a single structured light illumination image, and only uses one acquisition to improve the data acquisition speed and reduce motion artifacts. However, due to the different heights of the surface of complex morphology tissues, the optical response will be sensitive to the gradient changes caused by the height differences. In the measurement of complex morphology tissues, the edge demodulation effect due to height changes is not good. In 2021, Dan M et al. proposed a contour adaptive projection method to correct the spatial frequency deviation of single snapshot imaging. This method improves the accuracy of single snapshot imaging in extracting the optical characteristics of complex contour samples.

[0004] Therefore, the general process of the traditional spatial frequency domain imaging method is as follows: in the first stage, multiple spatial frequency domain images of the tissue to be tested are taken; in the second stage, the multiple spatial frequency domain images of the tissue to be tested are obtained by a three-phase demodulation algorithm to obtain the AC component M of the tissue to be tested. AC and the DC component M of the tissue DC ; In the third stage, the AC component M AC and DC component M DC u is obtained by diffusion theory and Monte Carlo simulation respectively. a and u s Optical parameter image; in the fourth stage, u a and u s The information of the physiological tissue to be tested is obtained through the Lambert-Beer equation.

[0005] However, the defects of the above-mentioned traditional spatial frequency domain imaging method in practical application are: in the first stage, whether it is collecting multiple spatial frequency domain images or a single snapshot image, the image contains the tissue area to be tested and other irrelevant areas. Since the spatial frequency domain imaging method demodulates the image point by point, the captured image will contain information that does not belong to the tissue to be tested. If the entire image is demodulated directly, the amount of calculation and time will increase. Since there is a certain switching time during the shooting process of different wavelengths, during this switching time, young and elderly people and subjects with underlying diseases may shake and move, which will cause an error in obtaining physiological information under different wavelength conditions, causing the image position to not match and thus affecting the experimental results.

[0006] In addition, by combining a line-scan hyperspectral camera with the SFDI system, continuous wavelength SFDI optical characteristic parameter detection can be achieved through the principle of hyperspectral imaging, and hyperspectral cameras can be used to perform hyperspectral imaging of tissues. However, hyperspectral cameras are expensive, resulting in a sharp increase in costs. Summary of the invention

[0007] The technical problem to be solved by the present invention is how to quickly obtain spatial frequency domain imaging with better effect.

[0008] The present invention solves the above technical problems by the following technical means:

[0009] A spatial frequency domain imaging method is proposed, the method comprising:

[0010] Collecting first optical parameter images of the tissue to be tested at different wavelengths;

[0011] Inputting the first optical parameter map at different wavelengths into a semantic segmentation network to obtain a second optical parameter map at different wavelengths that only includes the tissue to be tested;

[0012] Taking the second optical parameter map at one wavelength as a reference image, registering the second optical parameter maps at the remaining wavelengths to obtain registered optical parameter maps with correct relative positions at different wavelengths;

[0013] The registered optical parameter images at different wavelengths are merged into an RGB image, and the RGB image is input into a spectral reconstruction network to obtain a hyperspectral image of the tissue to be tested.

[0014] Furthermore, the collecting of first optical parameter images of the tissue to be tested at different wavelengths includes:

[0015] Single snapshot multi-frequency demodulation and three-dimensional morphology correction technology are used to collect the first optical parameter map of the tissue to be tested at different wavelengths.

[0016] Furthermore, the semantic segmentation network includes an encoding module and a decoding module, and the output of the encoding module is connected to the decoding module;

[0017] The encoding module includes a first encoding convolution unit, a second encoding convolution unit, a third encoding convolution unit, a fourth encoding convolution unit and a fifth encoding convolution unit which are connected in sequence and have increasing numbers of channels;

[0018] The first coded convolution unit includes a first convolution layer, a second convolution layer, and a first down-sampling layer that are connected in sequence and have the same number of channels, the second coded convolution unit includes a third convolution layer, a fourth convolution layer, and a second down-sampling layer that are connected in sequence and have the same number of channels, the third coded convolution unit includes a fifth convolution layer, a sixth convolution layer, and a third down-sampling layer that are connected in sequence and have the same number of channels, the fourth coded convolution unit includes a seventh convolution layer, an eighth convolution layer, and a fourth down-sampling layer that are connected in sequence and have the same number of channels, and the fifth coded convolution unit includes a ninth convolution layer and a tenth convolution layer that are connected in sequence and have the same number of channels.

[0019] Further, the decoding module includes a first decoding convolution unit, a second decoding convolution unit, a third decoding convolution unit, a fourth decoding convolution unit and a fifth decoding convolution unit which are connected in sequence and have decreasing number of channels;

[0020] The first decoding convolution unit comprises a first upsampling layer, and an input of the first upsampling layer is connected to an output of the tenth convolution layer;

[0021] The second decoding convolution unit includes a first stacked layer, an eleventh convolutional layer, a twelfth convolutional layer and a second upsampling layer connected in sequence, and the input of the first stacked layer is connected to the output of the eighth convolutional layer;

[0022] The third decoding convolution unit includes a second stacked layer, a thirteenth convolutional layer, a fourteenth convolutional layer and a third upsampling layer connected in sequence, and the input of the second stacked layer is connected to the output of the sixth convolutional layer;

[0023] The fourth decoding convolution unit includes a third stacked layer, a fifteenth convolutional layer, a sixteenth convolutional layer and a fourth upsampling layer connected in sequence, and the input of the third stacked layer is connected to the output of the fourth convolutional layer;

[0024] The fifth decoding convolution unit includes a fourth stacked layer, a seventeenth convolutional layer, an eighteenth convolutional layer and a nineteenth convolutional layer connected in sequence, and the input of the fourth stacked layer is connected to the output of the second convolutional layer.

[0025] Furthermore, the second optical parameter map at one wavelength is used as a reference image, and the second optical parameter maps at other wavelengths are registered to obtain the registered optical parameter maps with correct relative positions at different wavelengths, including:

[0026] Performing edge recognition on the tissue region to be measured on each of the second optical parameter images, and performing feature point detection on the edge of the tissue to be measured using a feature point detection algorithm to obtain key feature points;

[0027] Binary encoding the key feature points to form a descriptor;

[0028] Using a descriptor corresponding to the second optical parameter map at one wavelength as a reference descriptor, and matching the descriptors corresponding to the second optical parameter maps at the remaining wavelengths with the reference descriptor to determine matching point pairs;

[0029] estimating the homography matrix between the second optical parameter map and the reference image at the remaining wavelengths according to the matching point pairs;

[0030] According to the corresponding homography matrix, the second optical parameter map at the remaining wavelength is registered with the reference image to obtain the registered optical parameter map with correct relative position at different wavelengths.

[0031] Furthermore, before performing edge recognition of the tissue region to be measured on each of the second optical parameter images and performing feature point detection on the edge of the tissue to be measured using a feature point detection algorithm to obtain key feature points, the method further includes:

[0032] Performing smoothing processing on each of the second optical parameter maps, and then performing convolution with a Gaussian kernel to obtain a Gaussian kernel two-dimensional image corresponding to each of the second optical parameter maps;

[0033] Correspondingly, performing edge recognition of the tissue region to be measured on each of the second optical parameter images specifically includes: performing edge recognition of the tissue region to be measured on each of the Gaussian kernel two-dimensional images.

[0034] Further, the descriptor corresponding to the second optical parameter map at one wavelength is used as a reference descriptor, and the descriptors corresponding to the second optical parameter maps at other wavelengths are matched with the reference descriptor to determine the matching point pairs, including:

[0035] Based on the Hamming distance, a brute force matching method is used to match the descriptors corresponding to the second optical parameter graphs at other wavelengths with the reference descriptors to determine matching point pairs.

[0036] Further, the spectrum reconstruction network includes a first cascade convolution layer, a spectrum converter, and a second cascade convolution layer connected in sequence, and the first cascade convolution layer is jump-connected to the output of the second cascade convolution layer;

[0037] The spectrum converter includes an encoding network and a decoding network, the output of the encoding network is connected to the decoding network, the encoding network includes a first spectrum attention block, a second spectrum attention block and a third spectrum attention block, the first spectrum attention block is connected to the second spectrum attention block through a downsampling layer, and the second spectrum attention block is connected to the third spectrum attention block through a downsampling layer;

[0038] The decoding network includes a fourth spectrum attention block and a fifth spectrum attention block, the output of the third spectrum attention block is connected to the stacking layer via an upsampling layer, the output of the second spectrum attention block is connected to the stacking layer, and the stacking layer is connected to the input of the fourth spectrum attention block via a convolutional layer;

[0039] The output of the fourth spectrum attention block is connected to another stacked layer via an upsampling layer, the output of the first spectrum attention block is connected to the other stacked layer, and the other stacked layer is connected to the fifth spectrum attention block via another convolutional layer;

[0040] The output of the fifth spectral attention block is mapped to the input of the first cascade convolutional layer and then output to the second cascade convolutional layer.

[0041] Furthermore, each spectral attention block includes a first normalization layer, a multi-head self-attention module, a second normalization layer and a feedforward network, the output of the first normalization layer is connected to the multi-head self-attention module, and the output of the multi-head self-attention module is concatenated with the input of the first normalization layer as the input of the second normalization layer;

[0042] The output of the second normalization layer is connected to the input of the feedforward network, and the output of the feedforward network is concatenated with the input of the normalization layer to serve as the output of the spectral attention block.

[0043] Furthermore, the feedforward network includes a third cascade convolution, a depthwise separable convolution and a fourth cascade convolution connected in sequence; an activation function is connected after the third cascade convolution and the depthwise separable convolution;

[0044] The multi-head self-attention module includes three linear mapping operations Q, K and V in parallel. The linear mapping operations Q and K are input to the spectral self-attention head after multiplication. The output of the spectral self-attention head and the output of the linear mapping operation V are multiplied and then encoded through position PE embedding to encode the position information of different spectral channels.

[0045] The advantages of the present invention are:

[0046] (1) The present invention implements tissue region segmentation by inputting an optical parameter map of a tissue diffuse reflectance image into a semantic segmentation network, thereby reducing the amount of computation of the entire image to a key area, i.e., an area containing only the tissue to be measured, so that when demodulating the image point by point, it is avoided that pixels that do not belong to the tissue region to be measured need to be demodulated, thereby enabling rapid image demodulation; secondly, by performing key point registration on an optical parameter image obtained by spatial frequency domain imaging after shooting at different wavelengths, the relative positions of tissue regions in optical parameter images of different channels are matched, thereby improving the situation in which jitter affects the measurement results during multi-wavelength measurement, and increasing the accuracy of relative position information of demodulation results of different wavelengths after merging channels; furthermore, a spatial frequency domain imaging system measures optical parameters of multiple wavelengths and merges the channels to form an RGB image, and a corresponding hyperspectral image is obtained from the RGB image. In the spatial frequency domain imaging system, not only tissue parameters can be measured, but also richer tissue spectral information can be obtained.

[0047] (2) A single-snapshot multi-frequency demodulation algorithm and three-dimensional morphology correction are used to acquire diffuse reflectance images of the tissue to be measured, thus avoiding the problem of parameter map mismatch caused by jitter in multi-phase measurement of traditional spatial frequency domain imaging methods.

[0048] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 is a schematic flow chart of a spatial frequency domain imaging method proposed in one embodiment of the present invention;

[0050] Figure 2 is a schematic diagram of an experimental platform device in one embodiment of the present invention;

[0051] Figure 3 is a diagram showing the principle of semantic segmentation in one embodiment of the present invention;

[0052] Figure 4 is a schematic diagram of spectrum reconstruction principle in one embodiment of the present invention;

[0053] Figure 5 is a schematic diagram of the structure of a semantic segmentation network in one embodiment of the present invention;

[0054] Figure 6 is a schematic diagram of the structure of a spectrum reconstruction network in one embodiment of the present invention;

[0055] Figure 7 is a schematic diagram of the structure of an SST (spectrum converter) module in one embodiment of the present invention;

[0056] Figure 8 is a schematic diagram of the structure of a FFN (feed-forward network) in one embodiment of the present invention;

[0057] Fig. 9 is a schematic diagram of the structure of a SAB (spectral attention) module in one embodiment of the present invention;

[0058] Fig.10 It is a schematic diagram of the overall process of spatial frequency domain imaging in one embodiment of the present invention. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in combination with the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0060] like Figure 1 As shown, an embodiment of the present invention provides a spatial frequency domain imaging method, the method comprising the following steps:

[0061] S10, collecting first optical parameter images of the tissue to be tested at different wavelengths;

[0062] It should be noted that the present embodiment uses digital micromirror device (DMD) structured illumination to accurately control and change the incident light pattern on the sample surface. By using structured illumination containing multiple sinusoidal waves at the same time, two mutually orthogonal sinusoidal stripe structured lights are projected onto the tissue to be tested in a light-proof environment. The camera collects images reflected by the tissue surface under three wavelength conditions (such as Figure 2 As shown), the optical parameter map is obtained by inputting the demodulation into the algorithm.

[0063] S20, inputting the first optical parameter map at different wavelengths into a semantic segmentation network to obtain a second optical parameter map at different wavelengths that only includes the tissue to be tested;

[0064] Specifically, before using the semantic segmentation network for image segmentation, the annotated spatial frequency domain images are first used to train and improve the semantic segmentation network to obtain a semantic segmentation network with optimal segmentation accuracy. The optical parameter results at three wavelengths are input into the semantic segmentation network, and the three optical parameter images at three wavelengths are segmented respectively, and three optical parameter images containing only tissue areas are output, such as Figure 3 shown.

[0065] This embodiment implements the segmentation of tissue regions by inputting the optical parameter map into the semantic segmentation network, thereby reducing the computational complexity of the entire image to the key region, i.e., the region containing only the tissue to be tested. This avoids the need to demodulate pixels that do not belong to the tissue region to be tested when demodulating the image point by point, thereby enabling rapid image demodulation.

[0066] S30, using the second optical parameter map at one wavelength as a reference image, registering the second optical parameter maps at other wavelengths to obtain registered optical parameter maps with correct relative positions at different wavelengths;

[0067] This embodiment performs key point registration on the optical parameter images obtained by spatial frequency domain imaging after shooting at different wavelengths, so that the relative positions of tissue areas in the optical parameter images of different channels are matched, thereby improving the situation in which jitter affects the measurement results during multi-wavelength measurement, and increasing the accuracy of the relative position information after the demodulation results of different wavelengths are merged into channels.

[0068] S40, merging the channels of the registered optical parameter images at different wavelengths to form an RGB image, and inputting the RGB image into a spectral reconstruction network to obtain a hyperspectral image of the tissue to be measured.

[0069] Specifically, the three optical parameter images after registration are merged into an RGB image through an image processing algorithm. The spectral reconstruction network is improved by training paired RGB images and hyperspectral images to obtain the optimal spectral reconstruction network. The RGB image is input into the optimal spectral reconstruction network, and a hyperspectral image containing only the tissue area is output, such as Figure 4 shown.

[0070] It should be noted that in this embodiment, the spatial frequency domain imaging system measures the optical parameters of multiple wavelengths and merges the channels to form an RGB image, and the corresponding hyperspectral image is obtained from the RGB image. In the spatial frequency domain imaging system, not only can the tissue parameters be measured, but also richer tissue spectral information can be obtained.

[0071] As a further preferred technical solution, the step S10: collecting first optical parameter images of the tissue to be tested at different wavelengths includes:

[0072] Single snapshot multi-frequency demodulation and three-dimensional morphology correction technology are used to collect the first optical parameter map of the tissue to be tested at different wavelengths.

[0073] Specifically, the principle of single-shot multi-frequency demodulation and three-dimensional shape correction is: first, two mutually orthogonal sinusoidal stripe structured lights are projected onto the tissue to be tested, and the camera collects the light intensity information of the image reflected by the tissue surface. The amplitude M of each AC component is obtained by the integration method. AC,i , through demodulation, the DC component of the organization and the AC components in each direction are obtained.

[0074] In structured light projection, the stripes parallel to the plane where the projector and the optical collection axis are located are insensitive to height, while the stripes perpendicular to the plane where the projector and the optical collection axis are located are sensitive to height. The tissue phase in the insensitive direction can be approximately regarded as the phase of the reference plane, and the tissue phase in the sensitive direction can be regarded as the phase of the reference plane plus a phase difference.

[0075] Therefore, the phase of the reference plane in this direction is measured in advance, and the phase of the tissue to be measured is obtained from the relationship between the light intensity and the phase information. The phase difference can be obtained by unpacking the phase of the tissue to be measured and the phase of the reference plane and subtracting them. The phase difference and height use the triangular similarity of the system geometry to reconstruct the height profile of the sample. In this way, a linear relationship between the sample surface height and the phase difference is obtained.

[0076] For SFDI measurement, the light intensity is affected by the distance between the sample surface and the camera. The smaller the distance, the greater the intensity. In order to eliminate the influence of the sample height on the measurement, the system is height-calibrated. Similar to the phase height calibration, the optical properties of the diffuse reflector reference plane model are measured at n discrete heights. For each pixel, the light intensity changes accordingly with the change in height, and the change in light intensity is related to the AC component M. AC The change in M ​​is proportional to the change in AC The difference is used to calibrate the height difference.

[0077] It should be noted that the specific implementation process of the single-snapshot multi-frequency demodulation algorithm and three-dimensional morphology correction technology adopted in this embodiment can refer to the contents recorded in the application document with application publication number CN117968562A. By adopting the single-snapshot multi-frequency demodulation algorithm and three-dimensional morphology correction technology, real-time correction of three-dimensional morphology in single-snapshot multi-frequency demodulation is achieved, eliminating the influence of three-dimensional morphology on the results in structured light measurement, reducing the number of shooting times, and avoiding the problem of parameter map mismatch caused by jitter in multi-phase measurement of traditional spatial frequency domain imaging methods.

[0078] As a further preferred technical solution, Figure 5 As shown, the semantic segmentation network includes an encoding module and a decoding module, and the output of the encoding module is connected to the decoding module;

[0079] The encoding module includes a first encoding convolution unit, a second encoding convolution unit, a third encoding convolution unit, a fourth encoding convolution unit and a fifth encoding convolution unit which are connected in sequence and have increasing numbers of channels;

[0080] The first coded convolution unit includes a first convolution layer, a second convolution layer, and a first down-sampling layer that are connected in sequence and have the same number of channels, the second coded convolution unit includes a third convolution layer, a fourth convolution layer, and a second down-sampling layer that are connected in sequence and have the same number of channels, the third coded convolution unit includes a fifth convolution layer, a sixth convolution layer, and a third down-sampling layer that are connected in sequence and have the same number of channels, the fourth coded convolution unit includes a seventh convolution layer, an eighth convolution layer, and a fourth down-sampling layer that are connected in sequence and have the same number of channels, and the fifth coded convolution unit includes a ninth convolution layer and a tenth convolution layer that are connected in sequence and have the same number of channels.

[0081] Further, the decoding module includes a first decoding convolution unit, a second decoding convolution unit, a third decoding convolution unit, a fourth decoding convolution unit and a fifth decoding convolution unit which are connected in sequence and have decreasing number of channels;

[0082] The first decoding convolution unit comprises a first upsampling layer, and an input of the first upsampling layer is connected to an output of the tenth convolution layer;

[0083] The second decoding convolution unit includes a first stacked layer, an eleventh convolutional layer, a twelfth convolutional layer and a second upsampling layer connected in sequence, and the input of the first stacked layer is connected to the output of the eighth convolutional layer;

[0084] The third decoding convolution unit includes a second stacked layer, a thirteenth convolutional layer, a fourteenth convolutional layer and a third upsampling layer connected in sequence, and the input of the second stacked layer is connected to the output of the sixth convolutional layer;

[0085] The fourth decoding convolution unit includes a third stacked layer, a fifteenth convolutional layer, a sixteenth convolutional layer and a fourth upsampling layer connected in sequence, and the input of the third stacked layer is connected to the output of the fourth convolutional layer;

[0086] The fifth decoding convolution unit includes a fourth stacked layer, a seventeenth convolutional layer, an eighteenth convolutional layer and a nineteenth convolutional layer connected in sequence, and the input of the fourth stacked layer is connected to the output of the second convolutional layer.

[0087] Specifically, Figure 5As shown in Figure 2, when the input optical parameter map size is 512×512×3, the encoding module ( Figure 5 The processing process on the left side (from top to bottom) is:

[0088] The first encoding convolution unit first performs two Conv2d[3×3] convolutions with 64 channels to obtain the first feature layer f1[512,512,64], and then performs convolution downsampling with Conv2d[2×2] with the same number of channels.

[0089] The second encoding convolution unit performs two convolutions of Conv2d[3×3] with a channel number of 128 to obtain the second feature layer f2[256,256,128], and then performs convolution downsampling of Conv2d[2×2] with the same number of channels.

[0090] The third encoding convolution unit performs two Conv2d[3×3] convolutions with a channel number of 256 to obtain the third feature layer f3[128,128,256], and then performs convolution downsampling with Conv2d[2×2] with the same channel number.

[0091] The fourth coding convolution unit performs two convolutions of Conv2d[3×3] with a channel number of 512 to obtain the fourth feature layer f4[64,64,256], and then performs convolution downsampling of Conv2d[2×2] with the same number of channels.

[0092] The fifth encoding convolution unit performs two Conv2d[3×3] convolutions with a channel number of 1024 to obtain the fifth feature layer f5[32,32,1024].

[0093] The decoding process is ( Figure 5 Middle right side (from bottom to top):

[0094] The first decoding convolution unit first performs down-channeling of Conv2d[1×1] with a channel number of 512, and then up-samples.

[0095] The second decoding convolution unit is first stacked with the f4 feature layer, and then convolved twice with Conv2d[3×3] with a channel number of 512, and then upsampled.

[0096] The third decoding convolution unit is first stacked with the f3 feature layer, and then convolved twice with Conv2d[3×3] with a channel number of 256, and then upsampled.

[0097] The fourth decoding convolution unit is first stacked with the f2 feature layer, and then convolved twice with Conv2d[3×3] with a channel number of 128, and then upsampled.

[0098] The fifth decoding convolution unit is first stacked with the f1 feature layer, and then convolved twice with Conv2d[3×3] with 64 channels, and then upsampled.

[0099] Finally, a Conv2d[1×1] convolution with a channel number of 2 is performed to obtain a classification result of 2, which separates the area and the background.

[0100] It should be noted that, since it is necessary to keep the size of the spatial frequency domain image before and after segmentation unchanged, this embodiment uses gray bars to splice the image when inputting the image, so that the image can be scaled without distortion, and the input image size is resized to [512×512]. Since the size of the Unet network decreases after each convolution, the padding can be set to 1 to keep the size of the spatial frequency domain image unchanged during the convolution. In addition, adding BatchNorm to the convolution module can accelerate the training process and improve the generalization ability. At the same time, the ReLU activation function is replaced by the LeakyReLu activation function to avoid the disappearance of the gradient during training. In the upsampling network layer, due to the high similarity of pixels in the spatial frequency domain image, the nearest neighbor interpolation method is used to further restore the spatial frequency domain image.

[0101] As a further preferred technical solution, the step S30: using the second optical parameter map at one wavelength as a reference image, registering the second optical parameter maps at the remaining wavelengths to obtain the registered optical parameter maps with correct relative positions at different wavelengths, includes the following steps:

[0102] S31, performing edge recognition of the tissue region to be measured on each of the second optical parameter images, and performing feature point detection on the edge of the tissue to be measured using a feature point detection algorithm to obtain key feature points;

[0103] It should be noted that the optical parameter map in the spatial frequency domain image is a spatial frequency domain response map obtained at different wavelengths and has different modes, so conventional feature point detection cannot achieve good results. This feature detection uses a method that combines edge detection and corner point detection. First, the optical parameter map in the spatial frequency domain image is used to identify the edge of the tissue area to obtain the general target, and then the feature point detection algorithm is used to detect feature points on the edge.

[0104] First, the optical parameter image in the spatial frequency domain image is smoothed, and a Gaussian kernel is used for convolution to reduce a certain amount of noise to overcome the influence of noise on matching. The Gaussian kernel two-dimensional image formula is as follows:

[0105]

[0106] Where σ is the standard deviation of the Gaussian distribution, which is used to control the degree of blur smoothness, and x and y represent the distance of the image position relative to the center of the Gaussian kernel.

[0107] Then, the Canny algorithm is used to detect the edge of the optical parameter map, and the gradient amplitude is calculated using the Sobel operator. The calculation formula is as follows:

[0108]

[0109] In the formula, G x and G y are the gradients of the optical parameter image in the x and y directions.

[0110] Then, the ORB algorithm is used to detect the tissue edge in the optical parameter map, and in the FAST feature extraction, for each pixel point p, it is checked whether there are N consecutive pixels on the circumference of a circle that are larger or smaller than the pixel point p. If the condition is met, it is a feature point. In the ORB algorithm, it is proposed to use moments to determine the direction of FAST feature points. The moment is used to calculate the center of mass with r as the radius. The coordinates of the feature point to the center of mass form a vector as the direction. The moment is defined as follows:

[0111] m pq =∑ x,y x p a q I(x,y), x,y∈[-r,r]

[0112] The centroid expression is as follows:

[0113]

[0114] Feature point direction

[0115] In the formula, m pq is the moment, x p is the moment order of the x position, y q is the moment order of the y position, I(x,y) is the intensity of the image, r is the radius of the center of mass, m 00 is the moment when p=0,q=0,m 01 is the moment with p=0,q=1, m 10 is the moment with q=1,p=0.

[0116] S32, performing binary encoding on the key feature points to form a descriptor;

[0117] It should be noted that in the rBRIEF feature description, the selected feature points are binary encoded to form a descriptor.

[0118] S33, taking the descriptor corresponding to the second optical parameter map at one wavelength as a reference descriptor, and matching the descriptors corresponding to the second optical parameter maps at the remaining wavelengths with the reference descriptor to determine a matching point pair;

[0119] Specifically, the brute force matching method is used for feature matching based on the Hamming distance to calculate the similarity between two descriptors. The calculation formula is as follows:

[0120]

[0121] Where, d H (a,b) is the Hamming distance, a i is a binary string, b i is a binary string, It is an XOR operation.

[0122] S34, estimating the homography matrix H between the second optical parameter map and the reference image at the remaining wavelengths according to the matching point pairs;

[0123] S35. According to the corresponding homography matrix, the second optical parameter map at the remaining wavelength is registered with the reference image to obtain the registered optical parameter map with correct relative position at different wavelengths.

[0124] Specifically, this embodiment applies the estimated transformation model to realize the geometric transformation between images and complete the image registration. The transformation is as follows:

[0125]

[0126] Wherein, x′ is the x-coordinate position after conversion, y′ is the y-coordinate position after conversion, and ω′ is the ω-coordinate position after conversion.

[0127] This embodiment improves the results caused by jitter in multi-wavelength measurement of the traditional spatial frequency domain imaging method by adopting image registration, and increases the accuracy of relative position information after the demodulation results of different wavelengths are combined into channels.

[0128] As a further preferred technical solution, Figure 6 As shown, the spectrum reconstruction network includes a first cascade convolution layer, a spectrum converter, and a second cascade convolution layer connected in sequence, and the first cascade convolution layer is jump-connected to the output of the second cascade convolution layer;

[0129] The spectrum converter includes an encoding network and a decoding network, the output of the encoding network is connected to the decoding network, the encoding network includes a first spectrum attention block, a second spectrum attention block and a third spectrum attention block, the first spectrum attention block is connected to the second spectrum attention block through a downsampling layer, and the second spectrum attention block is connected to the third spectrum attention block through a downsampling layer;

[0130] The decoding network includes a fourth spectrum attention block and a fifth spectrum attention block, the output of the third spectrum attention block is connected to the stacking layer via an upsampling layer, the output of the second spectrum attention block is connected to the stacking layer, and the stacking layer is connected to the input of the fourth spectrum attention block via a convolutional layer;

[0131] The output of the fourth spectrum attention block is connected to another stacked layer via an upsampling layer, the output of the first spectrum attention block is connected to the other stacked layer, and the other stacked layer is connected to the fifth spectrum attention block via another convolutional layer;

[0132] The output of the fifth spectral attention block is mapped to the input of the first cascade convolutional layer and then output to the second cascade convolutional layer.

[0133] It should be noted that the input of the spectral reconstruction network is the RGB formed by merging channels of the registered optical parameter images obtained at multiple wavelengths. The main structure of the image spectral reconstruction network uses a cascade of 1 spectrum transformer (SST). The spectrum transformer adopts a U-shaped structure of encoding and decoding. By cascading two Conv (3×3) convolutions and an SST module, the final output is superimposed using jump connections to obtain the tissue hyperspectral image.

[0134] As a further preferred technical solution, Figure 7 As shown, each spectrum attention block SAB includes a first normalization layer, a multi-head self-attention module S-MSA, a second normalization layer and a feed-forward network FFN, the output of the first normalization layer is connected to the multi-head self-attention module, and the output of the multi-head self-attention module is concatenated with the input of the first normalization layer as the input of the second normalization layer;

[0135] The output of the second normalization layer is connected to the input of the feedforward network, and the output of the feedforward network is concatenated with the input of the normalization layer to serve as the output of the spectral attention block.

[0136] As a further preferred technical solution, Figure 8 As shown, the feedforward network includes a third cascade convolution, a depthwise separable convolution and a fourth cascade convolution connected in sequence; an activation function is connected after the third cascade convolution and the depthwise separable convolution.

[0137] It should be noted that two Conv(1×1) convolutions are cascaded, and the GELU activation function is used behind them. When the input value is small, it is close to 0, and when the input value is large, it is close to the input itself, which is conducive to processing data such as spatial frequency domain images. A depth-separable convolution Conv(3×3) is also inserted in the middle to convert the standard convolution into a depth-wise and successive convolution, thereby reducing the amount of calculation and the number of parameters.

[0138] As a further preferred technical solution, Fig. 9 As shown, the multi-head self-attention module includes three linear mapping operations Q, K and V in parallel. The linear mapping operations Q and K are input to the spectral self-attention head after multiplication. The output of the spectral self-attention head and the output of the linear mapping operation V are multiplied and then encoded through position PE embedding to encode the position information of different spectral channels.

[0139] Specifically, the working principle of the multi-head self-attention module S-MSA is as follows:

[0140] When the input is X in When , the data shape needs to be changed from [H, W, C] to [HW, C] before inputting the network. Then the input is linearly mapped to Q, K and V through three matrices. The formula is as follows:

[0141] Q=XW Q , K = XW K , V = XW V

[0142] Where Q is the embedding matrix, K is the embedding matrix, V is the embedding matrix, X is the input image with shape [HW, C], and W Q is the learnable parameter in training, W K is the learnable parameter in training, W V are learnable parameters during training.

[0143] S-MSA considers each spectral representation as a token and calculates the head j Attention, transpose the K matrix and multiply it with Q, assign weights through σ, and use softmax to get the self-attention coefficient A j , the self-attention coefficient A j Multiply it with V to get the corrected matrix, the formula is as follows:

[0144]

[0145] In the formula, σ j is the weight value, is the transposed matrix of K matrix, Q j is the Q matrix, V jis the V matrix, softmax() is the softmax function, head j For head attention.

[0146] The corrected spectral data are superimposed, and the position information of different spectral channels is encoded by position PE embedding. The formula is as follows:

[0147]

[0148] Where, X out is the output spectral image, W is the learnable parameter in training, N is the N heads divided into Q, K, and V, Concat() is the stacking operation, and f p () is the function of the embedding position.

[0149] The spectral reconstruction network designed in this embodiment is different from other spectral reconstruction methods that reconstruct hyperspectral images from naturally captured RGB images. It reconstructs richer tissue optical information from tissue optical parameters obtained from a small number of wavelengths.

[0150] In summary, if Fig.10 As shown, a method for measuring tissue parameters by spatial frequency domain imaging proposed in this embodiment, the first step is to obtain corrected optical parameter results by a single snapshot and three-dimensional morphology correction, the second step is to obtain optical parameter results containing only regional tissue by image segmentation, the third step is to align the relative positions of the optical parameter results by image registration and combine them into an RGB three-channel image, and the fourth step is to reconstruct a hyperspectral image of the tissue area from the RGB image after merging the channels; high-quality spatial frequency domain imaging can be quickly obtained.

[0151] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0152] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0153] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. A spatial frequency domain imaging method, characterized in that: The method comprises: Collecting first optical parameter images of the tissue to be tested at different wavelengths; Inputting the first optical parameter map at different wavelengths into a semantic segmentation network to obtain a second optical parameter map at different wavelengths that only includes the tissue to be tested; Taking the second optical parameter map at one wavelength as a reference image, registering the second optical parameter maps at the remaining wavelengths to obtain registered optical parameter maps with correct relative positions at different wavelengths; The registered optical parameter images at different wavelengths are merged into an RGB image, and the RGB image is input into a spectral reconstruction network to obtain a hyperspectral image of the tissue to be tested.

2. The spatial frequency domain imaging method according to claim 1, characterized in that: The collecting of first optical parameter images of the tissue to be tested at different wavelengths comprises: Single snapshot multi-frequency demodulation and three-dimensional morphology correction technology are used to collect the first optical parameter map of the tissue to be tested at different wavelengths.

3. The spatial frequency domain imaging method according to claim 1, characterized in that: The semantic segmentation network includes an encoding module and a decoding module, and the output of the encoding module is connected to the decoding module; The encoding module includes a first encoding convolution unit, a second encoding convolution unit, a third encoding convolution unit, a fourth encoding convolution unit and a fifth encoding convolution unit which are connected in sequence and have increasing numbers of channels; The first coded convolution unit includes a first convolution layer, a second convolution layer, and a first down-sampling layer that are connected in sequence and have the same number of channels, the second coded convolution unit includes a third convolution layer, a fourth convolution layer, and a second down-sampling layer that are connected in sequence and have the same number of channels, the third coded convolution unit includes a fifth convolution layer, a sixth convolution layer, and a third down-sampling layer that are connected in sequence and have the same number of channels, the fourth coded convolution unit includes a seventh convolution layer, an eighth convolution layer, and a fourth down-sampling layer that are connected in sequence and have the same number of channels, and the fifth coded convolution unit includes a ninth convolution layer and a tenth convolution layer that are connected in sequence and have the same number of channels.

4. The spatial frequency domain imaging method according to claim 3, characterized in that: The decoding module includes a first decoding convolution unit, a second decoding convolution unit, a third decoding convolution unit, a fourth decoding convolution unit and a fifth decoding convolution unit which are connected in sequence and have decreasing number of channels; The first decoding convolution unit comprises a first upsampling layer, an input of the first upsampling layer is connected to an output of the tenth convolution layer; The second decoding convolution unit includes a first stacked layer, an eleventh convolutional layer, a twelfth convolutional layer and a second upsampling layer connected in sequence, and the input of the first stacked layer is connected to the output of the eighth convolutional layer; The third decoding convolution unit includes a second stacked layer, a thirteenth convolutional layer, a fourteenth convolutional layer and a third upsampling layer connected in sequence, and the input of the second stacked layer is connected to the output of the sixth convolutional layer; The fourth decoding convolution unit includes a third stacked layer, a fifteenth convolutional layer, a sixteenth convolutional layer and a fourth upsampling layer connected in sequence, and the input of the third stacked layer is connected to the output of the fourth convolutional layer; The fifth decoding convolution unit includes a fourth stacked layer, a seventeenth convolutional layer, an eighteenth convolutional layer and a nineteenth convolutional layer connected in sequence, and the input of the fourth stacked layer is connected to the output of the second convolutional layer.

5. The spatial frequency domain imaging method according to claim 1, characterized in that: The method of using the second optical parameter map at one wavelength as a reference image and registering the second optical parameter maps at other wavelengths to obtain the registered optical parameter maps with correct relative positions at different wavelengths includes: Performing edge recognition on the tissue region to be measured on each of the second optical parameter images, and performing feature point detection on the edge of the tissue to be measured using a feature point detection algorithm to obtain key feature points; Binary encoding the key feature points to form a descriptor; Using a descriptor corresponding to the second optical parameter map at one wavelength as a reference descriptor, and matching the descriptors corresponding to the second optical parameter maps at the remaining wavelengths with the reference descriptor to determine matching point pairs; estimating the homography matrix between the second optical parameter map and the reference image at the remaining wavelengths according to the matching point pairs; According to the corresponding homography matrix, the second optical parameter map at the remaining wavelength is registered with the reference image to obtain the registered optical parameter map with correct relative position at different wavelengths.

6. The spatial frequency domain imaging method according to claim 5, characterized in that: Before performing edge recognition of the tissue region to be measured on each of the second optical parameter images and performing feature point detection on the edge of the tissue to be measured using a feature point detection algorithm to obtain key feature points, the method further includes: Performing smoothing processing on each of the second optical parameter maps, and then performing convolution with a Gaussian kernel to obtain a Gaussian kernel two-dimensional image corresponding to each of the second optical parameter maps; Correspondingly, performing edge recognition of the tissue region to be measured on each of the second optical parameter images specifically includes: performing edge recognition of the tissue region to be measured on each of the Gaussian kernel two-dimensional images.

7. The spatial frequency domain imaging method according to claim 5, characterized in that: The method of using a descriptor corresponding to the second optical parameter map at one wavelength as a reference descriptor, and matching the descriptors corresponding to the second optical parameter maps at other wavelengths with the reference descriptor to determine a matching point pair includes: Based on the Hamming distance, a brute force matching method is used to match the descriptors corresponding to the second optical parameter graphs at other wavelengths with the reference descriptors to determine matching point pairs.

8. The spatial frequency domain imaging method according to claim 1, characterized in that: The spectrum reconstruction network comprises a first cascade convolution layer, a spectrum converter and a second cascade convolution layer connected in sequence, wherein the first cascade convolution layer is jump-connected to the output of the second cascade convolution layer; The spectrum converter includes an encoding network and a decoding network, the output of the encoding network is connected to the decoding network, the encoding network includes a first spectrum attention block, a second spectrum attention block and a third spectrum attention block, the first spectrum attention block is connected to the second spectrum attention block through a downsampling layer, and the second spectrum attention block is connected to the third spectrum attention block through a downsampling layer; The decoding network includes a fourth spectrum attention block and a fifth spectrum attention block, the output of the third spectrum attention block is connected to the stacking layer via an upsampling layer, the output of the second spectrum attention block is connected to the stacking layer, and the stacking layer is connected to the input of the fourth spectrum attention block via a convolutional layer; The output of the fourth spectrum attention block is connected to another stacked layer via an upsampling layer, the output of the first spectrum attention block is connected to the other stacked layer, and the other stacked layer is connected to the fifth spectrum attention block via another convolutional layer; The output of the fifth spectral attention block is mapped to the input of the first cascade convolutional layer and then output to the second cascade convolutional layer.

9. The spatial frequency domain imaging method according to claim 8, characterized in that: Each spectral attention block includes a first normalization layer, a multi-head self-attention module, a second normalization layer and a feedforward network, wherein the output of the first normalization layer is connected to the multi-head self-attention module, and the output of the multi-head self-attention module is concatenated with the input of the first normalization layer as the input of the second normalization layer; The output of the second normalization layer is connected to the input of the feedforward network, and the output of the feedforward network is concatenated with the input of the normalization layer to serve as the output of the spectral attention block.

10. The spatial frequency domain imaging method according to claim 9, characterized in that: The feedforward network includes a third cascade convolution, a depthwise separable convolution and a fourth cascade convolution connected in sequence; an activation function is connected after the third cascade convolution and the depthwise separable convolution; The multi-head self-attention module includes three linear mapping operations Q, K and V in parallel. The linear mapping operations Q and K are input to the spectral self-attention head after multiplication. The output of the spectral self-attention head and the output of the linear mapping operation V are multiplied and then encoded through position PE embedding to encode the position information of different spectral channels.

Images