Method for accurately segmenting and registering multispectral tongue body image

Abstract

The invention relates to a method for accurate segmentation and registration of a multispectral tongue image, and belongs to the technical field of image processing. Firstly, a multispectral camera is used for shooting tongue body image data sets of a subject, a trained U-Net model is used for accurately segmenting each tongue body image in each group of data sets, and tongue body areas are extracted. Secondly, performing registration operation on the same group of tongue body images according to the segmented contour feature points, and then processing non-rigid deformation of fine textures in the tongue body by combining a dense optical flow method and a multi-stage registration strategy; and finally, enhancing the number and distribution of the control points by introducing free shape transformation (FFD) and an improved thin plate spline (TPS) algorithm. Experimental results show that color information of each image is reserved, the registration effect of tongue texture details is remarkably improved, and a reliable data basis is provided for subsequent tongue diagnosis analysis and multi-spectral information fusion.

Description

Technical Field

[0001] The invention relates to a method for accurate segmentation and registration of multispectral tongue images, belonging to the technical field of image processing, and in particular to a multispectral image registration and fusion technology for non-rigid objects. Background Art

[0002] Multispectral tongue diagnosis is an important method of TCM diagnosis. By analyzing the color and texture information of the tongue under different spectra, the health status of the subject can be obtained. However, as a non-rigid object, the tongue will undergo slight deformation in images taken in different time series, resulting in the inability to directly correspond to the color information at the same pixel position. This poses a challenge to the registration of multispectral tongue images. Traditional image registration methods have deficiencies in handling non-rigid deformations and subtle texture alignment, and cannot meet the requirements of multispectral tongue image analysis. Summary of the invention

[0003] The technical problem to be solved by the present invention is to provide a method for accurate segmentation and registration of multispectral tongue images. In the case of tongue deformation, traditional image registration methods are insufficient in processing non-rigid deformation and subtle texture alignment, and cannot meet the requirements of multispectral tongue image analysis.

[0004] The technical solution of the present invention is: a method for accurate segmentation and registration of multispectral tongue images, which is suitable for accurate registration and fusion of multispectral tongue images of subjects. By performing image segmentation, feature extraction, non-rigid registration and fusion, the multispectral color information of the same pixel position is finally obtained. The specific steps are:

[0005] Step 1: Use a multispectral camera to obtain the subject's tongue image dataset. Each dataset contains multiple tongue images of the same subject under different spectra.

[0006] Step 2: Use the trained U-Net model to segment each tongue image and extract the tongue area.

[0007] Step 3: Extract the tongue contour points and internal feature points in the extracted tongue area.

[0008] Step 4: Based on the contour points and internal feature points, perform non-rigid image registration.

[0009] Step 5: Fuse the registered images to obtain multispectral color information at the same pixel position.

[0010] The Step 3 is specifically as follows:

[0011] Step 3.1: Use feature detection algorithm to extract internal feature points in the tongue area.

[0012] Step 3.2: The feature detection algorithm includes SIFT, SURF or ORB algorithm.

[0013] The Step 4 is specifically as follows:

[0014] Step 4.1: First, perform global rigid registration to reduce large-scale displacement.

[0015] Step 4.2: Then perform local non-rigid registration to further align the local area of ​​the tongue.

[0016] Step 4.3: In non-rigid registration, dense optical flow is used to calculate the pixel-level motion field between the reference image and the target image.

[0017] Step 4.4: Free shape transformation (FFD) is used to perform fine deformation control on the tongue area based on B-spline mesh.

[0018] Step 4.5: Improve the thin plate spline (TPS) algorithm to improve the accuracy of registration by increasing the number of control points and optimizing their distribution.

[0019] The Step 5 is specifically as follows:

[0020] Step 5.1: During the fusion process, the color information of each image is retained to ensure the integrity of the multispectral data.

[0021] Step 5.2: Extract and integrate the color information of the registered multispectral image at the same pixel position.

[0022] The beneficial effects of the present invention are as follows: the present invention is suitable for accurate registration and fusion of multispectral tongue images. By combining the extraction of tongue contour points and internal feature points, the dense optical flow method and multi-level registration strategy are used to solve the problem of non-rigid deformation of the tongue in different time series. The introduction of free shape transformation and improved thin plate spline algorithm enhances the number and distribution of control points, and improves the accuracy and robustness of registration. Experimental results show that this method significantly improves the registration effect of tongue texture details while retaining the color information of each image, providing a reliable data basis for subsequent tongue diagnosis analysis and multispectral information fusion. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a flow chart of the steps of the present invention;

[0024] Figure 2 It is a tongue effect picture collected by the multi-spectral camera of the present invention;

[0025] Figure 3 This is a rendering of the present invention after cutting for use in experiments;

[0026] Figure 4 This is the effect diagram after the tongue image is segmented using the U-Net model in the present invention;

[0027] Figure 5 Schematic diagram of extracting tongue contour points according to the present invention;

[0028] Figure 6 Schematic diagram of the extraction and matching of tongue internal feature points of the present invention;

[0029] Figure 7 16 schematic diagrams of tongue registration under different spectra of the present invention;

[0030] Figure 8 This is a diagram of the fusion result of the multi-spectral tongue image after registration of the present invention. DETAILED DESCRIPTION

[0031] The present invention will be further described below in conjunction with the accompanying drawings and specific implementation methods.

[0032] Example 1: Figure 1 As shown, a method for accurate segmentation and registration of multispectral tongue images is provided, and the specific steps are as follows:

[0033] Step 1: Use a multispectral camera to obtain a dataset of the subject's tongue images. Each dataset contains tongue images of the same subject under 16 different spectra. Since the multispectral camera shoots in different time series and emits different spectra every second, the tongue, as a non-rigid object, may be slightly deformed.

[0034] Step 2: Use the trained U-Net model to segment each tongue image and extract the tongue area. Figure 2 As shown in the figure, after segmentation by the U-Net model, an accurate tongue mask is obtained, and background interference is removed, laying the foundation for subsequent feature extraction and alignment.

[0035] Step 3: Extract the tongue contour points and internal feature points in the extracted tongue area.

[0036] Step 3.1: Use edge detection algorithms (such as the Canny operator) to obtain the contour of the tongue and extract contour points. Figure 3 As shown, the contour points clearly depict the edge shape of the tongue.

[0037] Step 3.2: Use feature detection algorithms (such as SIFT, SURF or ORB) to extract internal feature points in the tongue area. To increase the number and stability of feature points, the tongue image can be texture enhanced (such as histogram equalization). Figure 4As shown, the extracted internal feature points are evenly distributed on the tongue surface, which helps to capture subtle texture differences.

[0038] Step 4: Based on the contour points and internal feature points, perform non-rigid image registration.

[0039] Step 4.1: First, perform global rigid registration to reduce large-scale displacement and rotation errors. Using the matched feature points, calculate the preliminary affine transformation matrix to align the target image to the reference image. Figure 5 As shown in Figure 3, after global registration, the overall positions of the two images are basically consistent.

[0040] Step 4.2: Then perform local non-rigid registration to handle the slight deformation of the tongue. The specific steps include:

[0041] Step 4.2.1: Calculate the pixel-level motion field between the reference image and the target image using the dense optical flow method to obtain subtle deformation information inside the tongue.

[0042] Step 4.2.2: Introduce free-form deformation (FFD) to perform fine deformation control on the tongue area based on B-spline mesh. Figure 7 As shown, the control mesh of FFD shows the deformation of the tongue area before and after deformation.

[0043] Step 4.2.3: Improve the thin plate spline (TPS) algorithm, increase the number of control points and optimize their distribution to improve the registration accuracy. Use the contour points and internal feature points as the control points of TPS to calculate the non-rigid deformation field.

[0044] Step 4.3: Apply the calculated deformation field to the target tongue image to complete the non-rigid registration. Figure 6 As shown, the texture and contour of the tongue are precisely aligned after registration.

[0045] Step 5: Fuse the registered images to obtain multispectral color information at the same pixel position.

[0046] Step 5.1: Keep the color information of each image to ensure the integrity of multispectral data.

[0047] Step 5.2: Fuse the 16 registered tongue images at the pixel level to obtain rich spectral features. Figure 8 As shown, the fusion result contains the complete information of the tongue under different spectra.

[0048] Step 6: Evaluate the registration results, and use indicators such as mutual information and structural similarity index (SSIM) to evaluate the registration quality. Experimental results show that the registered images have significant improvements in texture details and edge alignment.

[0049] Specifically, in Step 3, in order to improve the registration accuracy, not only the contour points of the tongue are extracted, but also the feature points inside the tongue are extracted using the feature detection algorithm. These feature points are used as control points in the non-rigid registration of Step 4 to help capture the subtle deformation of the tongue.

[0050] Specifically, a multi-level strategy of global rigid registration and local non-rigid registration is adopted. The global registration corrects large displacement and rotation errors, and the local registration targets the local deformation of the tongue and ensures the alignment of texture details.

[0051] Specifically, in non-rigid registration, FFD and improved TPS algorithms are introduced to enhance the number and distribution of control points, perform fine deformation control on the tongue area, avoid excessive distortion, and improve the robustness of registration.

[0052] Specifically, in the fusion process of Step 5, special attention is paid to preserving the color information of each image. This ensures the integrity of the multispectral data and provides reliable data support for subsequent tongue diagnosis analysis.

[0053] By using the method of the present invention, multispectral tongue images of 25 subjects were registered and fused. The results showed that the registered images were highly consistent in tongue contour and texture details, and the multispectral color information of the same pixel position could be accurately obtained. The present invention provides an efficient and accurate multispectral tongue image registration method, which solves the registration difficulties caused by the deformation of the tongue in different time series. By combining the extraction of tongue contour points and internal feature points, using dense optical flow method, multi-level registration strategy, free shape transformation and improved thin plate spline algorithm, accurate registration and fusion of tongue images are achieved, providing a reliable technical means for multispectral tongue diagnosis research.

[0054] The specific implementation modes of the present invention are described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the above implementation modes, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.

Claims

1. A method for accurate segmentation and registration of multispectral tongue images, characterized by: Step 1: Use a multispectral camera to obtain a tongue image dataset of the subject. Each dataset contains multiple tongue images of the same subject under different spectra. Step 2: Use the trained U-Net model to segment each tongue image and extract the tongue area; Step 3: Extract tongue contour points and internal feature points in the extracted tongue area; Step 4: Based on the contour points and internal feature points, non-rigid image registration is performed; Step 5: Fuse the registered images to obtain multispectral color information at the same pixel position; Step 6: Evaluate the fused images and use indicators such as mutual information or structural similarity index to evaluate the registration quality.

2. The method for accurate segmentation and registration of multispectral tongue images according to claim 1, characterized in that: The Step 3 is specifically as follows: Step 3.1: Extract internal feature points in the tongue area using feature detection algorithm; Step 3.2: The feature detection algorithm includes SIFT, SURF or ORB algorithm.

3. The method for accurate segmentation and registration of multispectral tongue images according to claim 1, characterized in that: The Step 4 is specifically as follows: Step 4.1: Combine dense optical flow method and multi-level registration strategy to handle the subtle non-rigid deformation inside the tongue; Step 4.2: Introduce free shape transformation and improved thin plate spline algorithm to enhance the number and distribution of control points and improve the accuracy and robustness of registration.

4. The method for accurate segmentation and registration of multispectral tongue images according to claim 3, characterized in that: The Step 4.1 is specifically as follows: Step 4.1.1: Perform global rigid registration to reduce large-scale displacement; Step 4.1.2: Perform local non-rigid registration to further align the local area of ​​the tongue.

5. The method for accurate segmentation and registration of multispectral tongue images according to claim 1, characterized in that: The free shape transformation in Step 4.2 is based on a B-spline grid to perform fine deformation control on the tongue area.

6. The method for accurate segmentation and registration of multispectral tongue images according to claim 3, characterized in that: The improved thin plate spline algorithm improves the accuracy of registration by increasing the number of control points and optimizing their distribution.

7. The method for accurate segmentation and registration of multispectral tongue images according to claim 1, characterized in that: The Step 5 specifically retains the color information of each image during the fusion process to ensure the integrity of the multispectral data.

Images