Tongue image feature analysis method and system

Abstract

The invention relates to the technical field of image processing, in particular to a tongue image feature analysis method and system. The method comprises the following steps: acquiring lingual surface multi-modal image data including a hyperspectral image, a visible light image and a dynamic tongue image; performing high-frequency component amplification processing on the visible light image and the dynamic tongue image; respectively extracting tongue regions from the hyperspectral image, the processed visible light image and the dynamic tongue image, and carrying out image registration on the extracted tongue regions; performing feature extraction on the registered tongue region in the multi-modal image data of the lingual surface by using a deep neural network to obtain a tongue spectral feature, a static tongue feature and a lingual fibrillation feature, and performing fusion to obtain fused tongue image feature data; and the deep neural network is adopted to analyze the fused tongue image feature data, and the traditional Chinese medicine analysis result of the current tongue image is predicted, so that the dependence on a complex network structure is reduced, and the fusion effect of the tongue features and the accuracy of the analysis result are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a tongue image feature analysis method and system. Background Art

[0002] Tongue diagnosis, as a non-invasive examination method, has become an important part of TCM diagnosis. TCM believes that the tongue is a mirror of the human body and is closely related to the internal organs and meridians. Therefore, changes in the tongue image can reveal the functional state of the internal organs, and thus can reflect the health of the human body through changes in the shape, color, and tongue coating area of ​​the tongue.

[0003] With the application of big data technology and the advancement of artificial intelligence algorithms, higher requirements are placed on the scientificity and objectivity of traditional Chinese medicine diagnostic methods. For this reason, objective research on tongue diagnosis has emerged. For example, Chinese Patent 202410668227.7, a tongue image detection method based on the fusion of RGB images and hyperspectral images, extracts spatial features from RGB images by designing a spatial-graph subnetwork for enhancing the global and local features of the perceived image, and extracts spectral features from hyperspectral images by designing a spectral subnetwork for capturing the long-range correlation of spectral features, and then performs weighted summation of spatial features and spectral features to obtain the final fusion probability, thereby processing the tongue image into category information, which contains more biological information and is conducive to the accurate classification of health status. However, the existing patent still has the following technical problems:

[0004] 1. In the frequency domain of natural images, the energy distribution is mainly concentrated in the low-frequency area, while the high-frequency components are relatively small. However, the sharp transitions and structural texture features of the tongue in the tongue image mainly exist in the high-frequency part. In order to enhance the perception of the global and local features and spectral features of the RGB image, the patent designs a complex spatial-graph sub-network and spectral sub-network, which contains multiple attention-dense blocks and transition layers. This results in a large amount of calculation during the network training process and is prone to overfitting. The complex network structure also makes it more sensitive to external factors such as tongue image noise and illumination changes, resulting in insufficient robustness of the model.

[0005] 2. Due to the different acquisition equipment and conditions, the acquired RGB images and hyperspectral images will inevitably have differences in resolution, viewing angle, lighting conditions, etc., which may easily produce erroneous local feature information or introduce artifacts during the fusion process, which will adversely affect the feature fusion effect and the accuracy of the analysis results;

[0006] 3. This patent only considers the spatial and spectral features of static images, and ignores the evaluation indicators related to the health status of the central nervous system, resulting in incomplete diagnostic information on the health status of the tongue and limiting the in-depth analysis of the disease mechanism by the neural network. Summary of the invention

[0007] In response to the above technical problems, the present invention proposes a tongue image feature analysis method and system, which aims to reduce the complexity of the network structure while strengthening the key detail features in the image through the optimized design of the image processing algorithm, and further improve the fusion effect of the tongue features and the accuracy of the analysis results by comprehensive consideration of the diagnostic information of the tongue health status.

[0008] In a first aspect, the present application provides a tongue image feature analysis method, comprising the following steps:

[0009] Acquire tongue surface multimodal image data, wherein the tongue surface multimodal image data includes a hyperspectral image, a visible light image, and a dynamic tongue image;

[0010] A high-frequency component amplification process is performed on the visible light image and the dynamic tongue image to obtain a processed visible light image and a dynamic tongue image;

[0011] Extracting the tongue region from the hyperspectral image, the processed visible light image and the dynamic tongue image respectively, and performing image registration on the extracted tongue region;

[0012] A deep neural network is used to extract features of the registered tongue regions in the hyperspectral image, visible light image and dynamic tongue image, respectively, to obtain tongue spectral features, static tongue features and tongue tremor features.

[0013] The tongue spectrum features, static tongue features and tongue tremor features are fused to obtain fused tongue image feature data;

[0014] A deep neural network is used to analyze the fused tongue image feature data and predict the TCM analysis results of the current tongue image.

[0015] In some embodiments, high-frequency component amplification processing is performed on the visible light image to obtain a processed visible light image, including:

[0016] The visible light image is converted into the frequency domain using the discrete Fourier transform algorithm to obtain the frequency domain matrix of the visible light image;

[0017] Creating a high-pass mask, and determining high-frequency components of the frequency domain matrix based on the high-pass mask;

[0018] Adding a phase shift to the high frequency component to obtain a frequency domain matrix after adding the phase shift;

[0019] The frequency domain matrix after adding the phase shift is inversely discrete Fourier transformed to obtain the processed visible light image.

[0020] In some embodiments, the “performing an inverse discrete Fourier transform on the frequency domain matrix after adding the phase shift to obtain a processed visible light image” includes:

[0021] Calculate the phase gradient of each frequency component in the frequency domain matrix after adding the phase shift along the horizontal direction and the vertical direction;

[0022] determining a phase change of each frequency component based on a phase gradient of each frequency component;

[0023] Comparing the phase change of each frequency component with a preset phase change threshold, and determining adjacent frequency components whose phase change exceeds the preset phase change threshold based on the comparison result;

[0024] constructing a weight matrix based on a set of adjacent frequency components whose phase changes exceed a preset phase change threshold;

[0025] The weight matrix is ​​smoothed using a two-dimensional Gaussian function;

[0026] Multiply the smoothed weight matrix by the frequency domain matrix after adding the phase shift to obtain the final frequency domain matrix;

[0027] The final frequency domain matrix is ​​inversely discrete Fourier transformed to obtain the processed visible light image.

[0028] In some embodiments, extracting the tongue region from the hyperspectral image, the processed visible light image, and the dynamic tongue image respectively includes:

[0029] a. Select hyperspectral image I D The image I of the Cth channel in D-C , according to the preset threshold, the image I of the Cth channel D-C The mask M corresponding to the tongue area is segmented out D ;

[0030] The image I of the Cth channel D-C The mask M of the tongue area D Point-to-point multiplication is performed to obtain the tongue area in the hyperspectral image;

[0031] b. The processed visible light image I V Convert from RGB space to YCbCr space and select image I of Cr channel V-Cr , according to the preset threshold from the image I of the Cr channel V-Cr The mask M corresponding to the tongue area is segmented out V ;

[0032] The Cr channel image I V-Cr The mask M of the tongue area V Multiply point by point to obtain the tongue area in the visible light image;

[0033] c. sequentially reading a number of frames of images from the processed dynamic tongue image to form a dynamic tongue image sequence;

[0034] Each frame of the dynamic tongue image sequence is converted from RGB space to YCbCr space, and the image I of the Cr channel is selected. B-Cr , according to the preset threshold from the image I of the Cr channel B-Cr The mask M corresponding to the tongue area is segmented out B ;

[0035] The Cr channel image I B-Cr The mask M of the tongue area B Multiply point by point to obtain the tongue area in the dynamic tongue image.

[0036] In some embodiments, a deep neural network is used to extract features of the registered tongue region in the visible light image to obtain static tongue features, including:

[0037] The registered tongue area in the visible light image is input into the convolutional neural network to extract deep features related to color, texture, shape, and local structure, which are recorded as static tongue features.

[0038] In some embodiments, the registered tongue region in the visible light image includes a tongue surface region image and a tongue base region image, and a deep neural network is used to extract features of the registered tongue region in the visible light image to obtain static tongue features, including:

[0039] The tongue surface area image and the tongue base area image are input into a two-branch convolutional neural network, and the two branches respectively extract features from the tongue surface area image and the tongue base area image, one branch extracts tongue surface depth features related to color, texture, shape, and local structure, and the other branch extracts tongue base depth features related to color, local structure, and tissue structure;

[0040] The tongue surface depth features and tongue base depth features are fused to obtain comprehensive tongue body features, which are recorded as static tongue features.

[0041] In some embodiments, a deep neural network is used to extract features of the registered tongue region in the dynamic tongue image to obtain tongue tremor features, including:

[0042] The Sobel operator is used to perform edge detection on the registered tongue area in the dynamic tongue image to obtain the tongue contour of each frame in the dynamic tongue image sequence.

[0043] Track the coordinates of the tongue contour of each frame image to obtain the tongue contour point set of each frame image;

[0044] Fitting the tongue contour point set of each frame image to obtain the tongue contour parameterized curve of each frame image;

[0045] Analyze the parameterized curves of tongue contours of all frame images to obtain the dynamic behavior data of tongue contours;

[0046] The dynamic behavior data of the tongue contour is input into the neural network for regression tasks, and the deep features related to the vibration amplitude, frequency, duration, and vibration location are extracted and recorded as tongue vibration features.

[0047] In some embodiments, a deep neural network is used to extract features of the tongue region after registration in the hyperspectral image to obtain tongue spectral features, including:

[0048] The aligned tongue area in the hyperspectral image is input into the three-dimensional convolutional neural network to extract the deep features related to the chemical composition of the tongue, which are recorded as the tongue spectral features.

[0049] In some embodiments, a deep neural network is used to analyze the fused tongue image feature data and predict the TCM analysis result of the current tongue image, including:

[0050] The fused tongue image feature data is input into the neural network for classification tasks to obtain the probability that the current tongue image belongs to different categories of TCM analysis results, and the category with the highest probability is selected as the prediction result of the current tongue image.

[0051] In a second aspect, the present application provides a tongue image feature analysis system, comprising:

[0052] An image acquisition module, used to acquire tongue surface multimodal image data, wherein the tongue surface multimodal image data includes a hyperspectral image, a visible light image, and a dynamic tongue image;

[0053] An image processing module is used to amplify the high-frequency components of the visible light image and the dynamic tongue image to obtain the processed visible light image and the dynamic tongue image, extract the tongue area from the hyperspectral image, the processed visible light image and the dynamic tongue image, and perform image registration on the extracted tongue area;

[0054] A feature extraction module is used to extract features of the registered tongue area in the hyperspectral image, the visible light image and the dynamic tongue image using a deep neural network to obtain tongue spectral features, static tongue features and tongue tremor features;

[0055] A feature fusion module is used to fuse tongue spectral features, static tongue features and tongue tremor features to obtain fused tongue image feature data;

[0056] The feature analysis module is used to analyze the fused tongue image feature data using a deep neural network and predict the TCM analysis results of the current tongue image.

[0057] The beneficial technical effects of the present invention include at least:

[0058] 1. A tongue image feature analysis method and system are adopted. Through the optimization design of the tongue surface multimodal image processing algorithm, only the high-frequency component is added with a phase shift in the frequency domain. Through the phase adjustment of the high-frequency part, the key detail features in the tongue surface multimodal image data are greatly enhanced, and the visual effect of the visible light image is improved. Therefore, even if the present application adopts a general deep neural network, it can still more accurately learn and identify the pathological characteristics of the tongue image, thereby improving the classification or recognition accuracy of the final TCM analysis results, reducing the dependence on complex network structures, and reducing the calculation amount of the deep neural network, so that the neural network is more stable and reliable in the actual tongue image feature analysis situation. While extracting static tongue features and tongue spectral features, the tongue tremor feature, an important indicator related to the health status of the central nervous system, is taken into account, so that the diagnostic information of the health status of the tongue is fully considered, which is helpful to improve the fusion effect of the tongue features and the accuracy of the analysis results;

[0059] 2. The visible light image is converted to the frequency domain by using the discrete Fourier transform algorithm. In the frequency domain, only the high-frequency components are phase shifted, which is equivalent to changing the phase of these high-frequency components in the spatial domain. By adjusting the phase of the high-frequency part, the details of the tongue image (such as edges and textures) are enhanced, making the visible light image clearer and improving the visual effect of the visible light image. Furthermore, in order to highlight the high-frequency part, a weighting scheme based on the phase change between adjacent frequency domain components is proposed, and a two-dimensional Gaussian function is used to smooth the weight matrix, so as to avoid the tongue image from having artifacts due to the drastic changes in the weight matrix, so that the weight matrix after smoothing is It can effectively amplify high-frequency components to enhance the suppressed key details and edge information in high-frequency components, perform inverse discrete Fourier transform on the final frequency domain matrix after weighted processing based on phase change between adjacent frequency domain components, and convert the image after phase modification back to the spatial domain. The processed visible light image obtained in this way contains further prominent high-frequency components, thereby further improving the visual effect of the visible light image, and then perform image registration after extracting the tongue area in the three images, which helps to obtain more accurate feature vectors in the feature extraction stage, thereby greatly reducing the adverse effects on the subsequent feature fusion effect and the accuracy of the analysis results.

[0060] 3. Usually, the visible light image is divided into three channels of RGB. However, in the visible light image of the tongue, the difference between the tongue area and the background area is not obvious in the three RGB channels, and the image in the RGB space is greatly affected by the illumination. Therefore, the present application converts the visible light image and the tongue dynamic image from the RGB space to the YCbCr space, so that the difference between the tongue area and the background area has a significant expression, and selects the image of the Cr channel (i.e., the red channel) in the YCbCr space as the basis for tongue area extraction. Since the color of the tongue is usually reddish, the contrast between the tongue area and the background area is often higher in the Cr channel, making the tongue area more prominent. Moreover, the Cr channel is more adaptable to different illumination conditions and background colors, so the tongue area can be better extracted in different environments, which helps to improve the accuracy of tongue area extraction in visible light images and tongue dynamic images, and reduce the situation of mis-segmentation and missed segmentation.

[0061] 3. For the tongue tremor feature, an important indicator related to the health status of the central nervous system, a quantitative feature extraction method is specifically designed. Since the dynamic tongue image is continuously changing, the position and shape of the tongue contour in each frame image may be different. In order to better track the change of the tongue contour, this application uses the Sobel operator for edge detection, expands the image gradient of the tongue area in the horizontal and vertical directions, and obtains the change intensity of the tongue contour in two directions. After adding the gradient information, the moving path of the tongue contour can be more accurately predicted and tracked, so as to accurately extract the tongue contour, track the coordinates of the tongue contour line extracted from each frame image, continuously record the changes of the tongue contour and fit the tongue contour point set into a parameterized curve, and study the dynamic behavior data of the tongue contour by analyzing the changes in the parameterized curve of the time series. The neural network used for regression tasks is used to extract the deep tongue tremor features related to the tremor amplitude, frequency, duration, and tremor location, and is used together with the static tongue features and the hyperspectral features in the subsequent analysis of the fused tongue image feature data, so as to achieve comprehensive consideration of the diagnostic information of the tongue health status, which helps to improve the fusion effect of the tongue features and the accuracy of the analysis results;

[0062] 4. Since there are significant visual differences between the visible light image of the tongue surface and the visible light image of the tongue base, for example, the tongue surface usually has more texture and local structural details, such as tongue coating, cracks, tongue veins, etc., while the local structure of the tongue base image may be more complex, including the tissue structure of the tongue base, possible lesion areas, etc., therefore, this application uses different convolutional neural network branch architectures for the tongue surface and tongue base images to extract their respective local features, which can better capture the local structure and texture information of the tongue surface and tongue base images, and provide richer feature representation for subsequent classification or regression tasks. At the same time, the type of lesions that may occur in the tongue base area is different from that in the tongue surface area. Therefore, the design of the dual-branch convolutional neural network in this application can better adapt to these differences and improve the ability to predict or identify specific lesion types.

[0063] Other features and advantages of the present invention will be disclosed in detail in the following specific embodiments and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] The present invention will be further described below in conjunction with the accompanying drawings:

[0065] Figure 1 Flow chart of the tongue image feature analysis method according to an embodiment of the present invention.

[0066] Figure 2 Schematic diagram of the structure of the tongue image feature analysis system according to an embodiment of the present invention. DETAILED DESCRIPTION

[0067] The technical solutions of the embodiments of the present invention are explained and described below in conjunction with the drawings of the embodiments of the present invention, but the following embodiments are only preferred embodiments of the present invention, not all. Based on the embodiments in the implementation mode, other embodiments obtained by those skilled in the art without creative work are all within the protection scope of the present invention.

[0068] In the following description, terms such as "inside", "outside", "up", "down", "left", "right", etc. that indicate directions or positional relationships are only used to facilitate the description of the embodiments and simplify the description, and do not indicate or imply that the referred device or element must have a specific direction, be constructed and operate in a specific direction. Therefore, it should not be understood as a limitation of the present invention.

[0069] Please refer to the attached Figure 1 , Figure 1 A schematic flow chart of a tongue image feature analysis method provided in one embodiment of the present specification is shown.

[0070] like Figure 1 As shown, the tongue image feature analysis method may at least include the following steps:

[0071] S101, acquiring tongue surface multimodal image data, where the tongue surface multimodal image data includes a hyperspectral image, a visible light image, and a dynamic tongue image.

[0072] Among them, a hyperspectral image is an image that can capture the reflectance spectral information of the face and tongue in multiple continuous spectral bands (usually from ultraviolet to near-infrared). A hyperspectral image is a three-dimensional data structure, usually including two-dimensional spatial coordinates (or pixel positions) and wavelengths. The wavelength corresponds to the spectral intensity value of each pixel point, usually in nanometers (nm).

[0073] The dynamic tongue image may be a captured dynamic tongue video, or may be composed of a plurality of tongue images captured in sequence at fixed time intervals t by a dynamic image acquisition camera.

[0074] Furthermore, in order to improve the effectiveness of the acquired multimodal image data of the tongue surface, the method for acquiring the multimodal image data of the tongue surface may further include the following steps:

[0075] Acquire facial images of human faces in real time;

[0076] Using facial key point detection algorithm, obtain the coordinates of key points around the mouth in the face image;

[0077] Calculate the distance between the key point coordinates and compare it with the preset threshold. If the distance exceeds the preset threshold, it is determined that the mouth is open and the next step is executed:

[0078] In the image with the mouth open, threshold segmentation is performed on the inner area of ​​the mouth to extract the possible tongue area;

[0079] Calculate the area of ​​the tongue or count the number of pixels in the tongue area, and compare it with a preset threshold. If the calculated result exceeds the preset threshold, it is determined that the tongue is extended;

[0080] When the judgment result satisfies the condition that the mouth is open and the tongue is extended, the image acquisition module is triggered to obtain the required hyperspectral image, visible light image and dynamic tongue image to form multimodal image data of the tongue surface.

[0081] By detecting facial key points in real time and judging the mouth opening state and tongue extension state based on preset thresholds to trigger the collection of multimodal image data of the tongue surface, the effectiveness of the collection of multimodal image data of the tongue surface can be effectively improved, thereby improving the overall efficiency of tongue image feature analysis.

[0082] S102, performing high frequency component amplification processing on the visible light image and the dynamic tongue image to obtain processed visible light image and dynamic tongue image.

[0083] In the frequency domain of natural images, the energy distribution is mainly concentrated in the low-frequency area, while the high-frequency components are relatively small. The sharp transitions and structured texture features of the tongue in the tongue image mainly exist in the high-frequency part. Therefore, this embodiment proposes an optimization algorithm for multimodal image processing of the tongue surface, aiming to effectively enhance the key details in the tongue image.

[0084] Specifically, in this embodiment, high-frequency component amplification processing is performed on the visible light image to obtain a processed visible light image, including:

[0085] S201, using a discrete Fourier transform algorithm to convert the visible light image into a frequency domain, and obtaining a frequency domain matrix of the visible light image.

[0086] Let the original visible light image be x[m,n], first use the discrete Fourier transform (DFT) algorithm to convert the original visible light image from the spatial domain to the frequency domain, and obtain the frequency domain matrix X[u,v] of the visible light image as shown in the following formula (0.1). The size of the frequency domain matrix is

[0087]

[0088] Among them, |X[u,v]| is the modulus of X[u,v], that is, the amplitude spectrum of the frequency component, which represents the signal strength or energy size at the position (u,v) in the frequency domain matrix. jφ[u,v] is the phase spectrum of X[u,v], where φ[u,v] is the phase angle, which determines the phase information of each frequency component.

[0089] S202: Create a high-pass mask, and determine the high-frequency components of the frequency domain matrix based on the high-pass mask.

[0090] For example, the expression of the high-pass mask in this embodiment is shown in the following formula (0.2):

[0091]

[0092] Among them, u c and v c Next, the high-pass mask is multiplied by the frequency domain matrix, so that only high-frequency components are retained and low-frequency components are weakened or eliminated.

[0093] S203, adding a phase shift to the high frequency component to obtain a frequency domain matrix after adding the phase shift.

[0094] It is understandable that the low-frequency components in the frequency domain correspond to the smooth areas in the visible light image, and adding a phase shift θ to these low-frequency components may cause artifacts to appear in the smooth areas of the image, thereby reducing the quality of the visible light image. To this end, this embodiment only adds a phase shift to the high-frequency components in the frequency domain, which is equivalent to changing the phase of these high-frequency components in the spatial domain. By adjusting the phase of the high-frequency part, the details of the tongue image (such as edges and textures) are enhanced, making the visible light image clearer and improving the visual effect of the visible light image.

[0095] After adding the phase shift θ, the spectrum of the visible light image becomes as shown in the following equation (0.3):

[0096] X p [u,v]=|X[u,v]|e j(φ[u,v]+H[u,v]θsgn(φ[u,v])) (0.3)

[0097] Here, sgn is an operator used to preserve the conjugate symmetry of the discrete Fourier transform.

[0098] S204, performing an inverse discrete Fourier transform on the frequency domain matrix after adding the phase shift, to obtain a processed visible light image.

[0099] It can be understood that the frequency domain matrix after adding the phase shift is subjected to an inverse discrete Fourier transform (IDFT) and converted back to the spatial domain. The processed visible light image thus obtained contains the amplified high-frequency components, because the influence of the low-frequency components has been weakened, while the high-frequency details are retained or even enhanced, thereby effectively enhancing the image contrast and enhancing the key details and edge information in the image.

[0100] It can be understood that the method for amplifying the high-frequency component of the dynamic tongue image in this embodiment is: reading the dynamic tongue image frame by frame in sequence to obtain a dynamic tongue image sequence, and then applying the same high-frequency component amplification processing steps S201-S204 as the visible light image to the dynamic tongue image sequence to obtain the processed dynamic tongue image, which will not be repeated in this embodiment.

[0101] Furthermore, in order to highlight the high-frequency part, this embodiment proposes a weighting scheme based on the phase change between adjacent frequency domain components. Specifically, in this embodiment, "performing an inverse discrete Fourier transform on the frequency domain matrix after adding the phase shift to obtain a processed visible light image" includes:

[0102] S2041, calculating the phase gradient of each frequency component in the frequency domain matrix after adding the phase shift along the horizontal direction and the vertical direction.

[0103] It can be understood that in this embodiment, the phase change is obtained by calculating the phase gradient of each frequency component in the frequency domain matrix along the horizontal direction and the vertical direction. The calculation of is shown in the following formula (0.4):

[0104]

[0105] S2042: Determine the phase change of each frequency component based on the phase gradient of each frequency component.

[0106] For the phase gradient of each frequency component, the amplitude of its phase gradient is calculated to represent the magnitude of the phase change, that is, the phase change of each frequency component.

[0107] S2043, comparing the phase change of each frequency component with a preset phase change threshold, and determining adjacent frequency components whose phase change exceeds the preset phase change threshold based on the comparison result.

[0108] Among them, the preset phase change threshold in this embodiment is used to distinguish obvious phase changes from insignificant phase changes. The preset phase change threshold can be fixed or an adaptive threshold based on the tongue image content, which is not limited in this embodiment.

[0109] For example, the phase change of each frequency component is compared with a preset phase change threshold, and the adjacent frequency components whose phase change exceeds the preset phase change threshold are determined based on the comparison result as follows: each frequency component in the frequency domain matrix is ​​traversed, and the phase change of each frequency component is compared with the preset phase change threshold. If the phase change of a frequency component is greater than the preset phase change threshold, the phase change of the adjacent frequency components (frequency components in the horizontal and vertical directions) of the frequency component is checked. If the phase change of the adjacent frequency components is also greater than the preset phase change threshold, then the two frequency components are adjacent frequency components with obvious phase change.

[0110] S2044: construct a weight matrix based on a set of adjacent frequency components whose phase changes exceed a preset phase change threshold.

[0111] For example, the weight matrix W based on the phase transition p (u,v), as shown in equation (2.5):

[0112]

[0113] Among them, α is a preset proportional factor, S indicates that the phase change exceeds the preset phase change threshold τ φ A collection of adjacent frequency components.

[0114] S2045, using a two-dimensional Gaussian function to smooth the weight matrix.

[0115] This embodiment uses a two-dimensional Gaussian function g[u,v] to transform the weight matrix W P [u,v] is smoothed to avoid the tongue image being affected by the weight matrix W P The phenomenon of artifacts due to the drastic change of [u,v]. Let the weight matrix after smoothing be W f [u,v], which is defined as shown in the following formula (2.6):

[0116]

[0117] Among them, “*” is the convolution operator.

[0118] S2046, multiplying the smoothed weight matrix by the frequency domain matrix after adding the phase shift to obtain a final frequency domain matrix.

[0119] It can be understood that the smoothed weight matrix can effectively amplify the high-frequency components to enhance the key details and edge information that are suppressed in the high-frequency components.

[0120] S2047, performing an inverse discrete Fourier transform on the final frequency domain matrix to obtain a processed visible light image.

[0121] It can be understood that the final frequency domain matrix after weighted processing based on the phase change between adjacent frequency domain components is subjected to an inverse discrete Fourier transform, and the image after phase modification is converted back to the spatial domain. The processed visible light image obtained in this way contains further prominent high-frequency components, thereby further improving the visual effect of the visible light image.

[0122] S103, extracting the tongue region from the hyperspectral image, the processed visible light image, and the dynamic tongue image, respectively, and performing image registration on the extracted tongue region.

[0123] Specifically, in this embodiment, the tongue region is extracted from the hyperspectral image, the processed visible light image, and the dynamic tongue image, respectively, including:

[0124] a. Select hyperspectral image I D The image I of the Cth channel in D-C , according to the preset threshold, the image I of the Cth channel D-C The mask M corresponding to the tongue area is segmented out D ;

[0125] The image I of the Cth channel D-C The mask M of the tongue area D Point-to-point multiplication is performed to obtain the tongue area in the hyperspectral image.

[0126] It can be understood that in the hyperspectral image, different channels represent different wavelengths, and each channel can provide different information about the hyperspectral image of the tongue surface. In this embodiment, the selection of the Cth channel of the hyperspectral image of the tongue surface depends on the following factors:

[0127] 1. Contrast: Select a channel with high contrast between the tongue area and the background (such as the rest of the mouth, teeth, etc.), which means that the reflection or emission characteristics of the tongue area are obviously different from other areas on this channel.

[0128] 2. Tongue features: An ideal channel may better highlight specific features of the tongue surface, such as blood vessels, tongue coating, color changes, etc.

[0129] 3. Noise level: Selecting channels with lower noise can help improve the accuracy of threshold segmentation.

[0130] Preferably, this embodiment selects the image of the near-infrared channel of the hyperspectral image to extract the tongue area, because usually, the near-infrared channel can better penetrate the skin, highlight features such as blood vessels, and is more suitable for extracting the tongue area.

[0131] b. The processed visible light image I V Convert from RGB space to YCbCr space and select image I of Cr channel V-Cr , according to the preset threshold from the image I of the Cr channel V-Cr The mask M corresponding to the tongue area is segmented out V ;

[0132] The Cr channel image I V-Cr The mask M of the tongue area V Point-to-point multiplication is performed to obtain the tongue area in the visible light image.

[0133] It can be understood that, usually, the visible light image is divided into three channels of RGB. However, in the visible light image of the tongue, the difference between the tongue area and the background area is not obvious in the three RGB channels, and the image in the RGB space is greatly affected by the illumination. Therefore, the present embodiment converts the visible light image from the RGB space to the YCbCr space, so that the difference between the tongue area and the background area is significantly expressed, and selects the image of the Cr channel (i.e., the red channel) in the YCbCr space as the basis for extracting the tongue area. Since the color of the tongue is usually reddish, the contrast between the tongue area and the background area is often higher in the Cr channel, thereby making the tongue area more prominent. Moreover, the Cr channel is more adaptable to different illumination conditions and background colors. Therefore, the tongue area can be better extracted in different environments, which helps to improve the accuracy of tongue area extraction in visible light images and reduce mis-segmentation and missed segmentation.

[0134] c. sequentially reading a number of frames of images from the processed dynamic tongue image to form a dynamic tongue image sequence;

[0135] Each frame of the dynamic tongue image sequence is converted from RGB space to YCbCr space, and the image I of the Cr channel is selected. B-Cr , according to the preset threshold from the image I of the Cr channel B-Cr The mask M corresponding to the tongue area is segmented out B ;

[0136] The Cr channel image I B-Cr The mask M of the tongue area B Multiply point by point to obtain the tongue area in the dynamic tongue image.

[0137] Using the same tongue region extraction method for dynamic tongue images as for visible light images can also help improve the accuracy of tongue region extraction for each frame in the dynamic tongue image sequence and reduce mis-segmentation and missed segmentation.

[0138] Furthermore, in order to extract the image components most relevant to expressing and depicting tongue features from the hyperspectral image, the processed visible light image and the dynamic tongue image, so that the subsequent feature extraction work can grasp the most essential shape features of the tongue, the mask of the tongue area can also be morphologically processed, and the largest closed-loop area is taken as the tongue area to eliminate the noise caused by skin plaques. At the same time, the dilation operation is used to further optimize the mask morphology of the tongue area.

[0139] For example, in this embodiment, the image registration of the extracted tongue area may be implemented as follows:

[0140] a. For M B and M D Zoom in and get the zoomed and where β B and β D is the scaling factor obtained by pre-calibration;

[0141] b.Yes and M V Perform registration, consider the translation transformation, and obtain the deviation values ​​in two directions of the translation transformation by traversing Since each element in the mask is a logical value, the logical AND operation can be used to speed up the calculation;

[0142] c.Yes and M V The deviation value (Δx D ,ΔyD );

[0143] d. According to the pre-calibrated scaling factor and the calculated deviation value, the tongue area in the dynamic tongue image and the tongue area in the hyperspectral image are scaled and translated accordingly to finally obtain the registered tongue area.

[0144] S104, using a deep neural network to extract features of the registered tongue regions in the hyperspectral image, the visible light image, and the dynamic tongue image, respectively, to obtain tongue spectral features, static tongue features, and tongue tremor features.

[0145] It can be understood that in this embodiment, through the optimization design of the image processing algorithm in the above steps S101-S103, the key detail features in the multimodal image data of the tongue surface have been greatly enhanced. Therefore, even if the present embodiment adopts a general deep neural network, it can still more accurately learn and identify the pathological characteristics of the tongue image, thereby improving the classification or recognition accuracy of the final TCM analysis results, reducing the dependence on complex network structures, and reducing the computational complexity of the deep neural network, making the neural network more stable and reliable in actual tongue image feature analysis situations. While extracting static tongue features and tongue spectral features, the tongue tremor feature, an important indicator related to the health status of the central nervous system, is taken into account, thereby achieving comprehensive consideration of the diagnostic information on the health status of the tongue, which helps to improve the fusion effect of tongue features and the accuracy of the analysis results.

[0146] Specifically, in this embodiment, a deep neural network is used to extract features of the registered tongue region in the visible light image to obtain static tongue features, including:

[0147] The registered tongue area in the visible light image is input into the convolutional neural network to extract deep features related to color, texture, shape, and local structure, which are recorded as static tongue features.

[0148] Preferably, in this embodiment, for the visible light image of the tongue surface, a convolutional neural network architecture of ResNet (residual network) or DenseNet (densely connected network) may be considered. The specific reasons are as follows:

[0149] On the one hand, ResNet can effectively solve the gradient vanishing problem of deep networks. Its residual module helps the network maintain a high feature transfer capability when extracting image details at a deep level, effectively improving the training speed. It is suitable for extracting texture, cracks and other details in tongue images. By increasing the network depth, ResNet can capture higher-level image information;

[0150] On the other hand, DenseNet can effectively retain multi-layer information and ensure efficient transmission of features by densely connecting each layer with all previous layers. It is especially suitable for extracting delicate tongue texture features, which helps to improve the expressiveness of static tongue features and the stability of the network.

[0151] Among them, in this embodiment, the convolutional neural network extracts the deep features related to color, texture, shape, and local structure, and the specific implementation method is as follows:

[0152] 1. Color features: Since this embodiment selects the image of the Cr channel and focuses on extracting features related to color information, especially information related to red, the convolutional neural network can more easily capture the color distribution and changes in the tongue area.

[0153] 2. Texture features: The convolutional layer of the convolutional neural network can effectively extract the texture information of the tongue area, such as the roughness of the tongue coating, the shape and direction of the cracks, etc.

[0154] 3. Shape features: Through the combination of convolutional layers and pooling layers, the convolutional neural network can recognize the contours, edges, and shape features of the tongue, such as the overall shape of the tongue, the contours of the tongue border, etc.

[0155] 4. Local structural features: Convolutional neural networks can learn local structural features such as the distribution of tongue veins, the local pattern of tongue coating, the specific morphology of cracks, and capture the spatial relationship between tongue features, such as the relative position of tongue veins and tongue coating, the distribution of cracks, etc.

[0156] Specifically, in this embodiment, a deep neural network is used to extract features of the registered tongue region in the dynamic tongue image to obtain tongue tremor features, including:

[0157] S301, using the Sobel operator to perform edge detection on the registered tongue region in the dynamic tongue image, to obtain the tongue contour of each frame in the dynamic tongue image sequence.

[0158] The purpose of edge detection is to identify points in the tongue area where brightness changes significantly, and these points form the outline of the tongue.

[0159] Since the dynamic tongue image changes continuously, the position and shape of the tongue contour in each frame may be different. In order to better track the change of the tongue contour, this embodiment uses the Sobel operator to expand the image gradient of the tongue area in the horizontal and vertical directions to obtain the change intensity of the tongue contour in two directions. After adding the gradient information, the movement path of the tongue contour can be more accurately predicted and tracked.

[0160] S302, coordinate tracking of the tongue contour of each frame image is performed to obtain a tongue contour point set of each frame image.

[0161] Specifically, the coordinates of the tongue contour extracted from each frame of the image are tracked, and the coordinate position of each point on the contour is recorded. These coordinate point sets constitute the tongue contour point set, which represents the precise position of the tongue contour. The tracked contour point set can be saved in the form of a list, array or other data structure for subsequent shape analysis, geometric measurement or other forms of analysis.

[0162] S303, fitting the tongue contour point set of each frame image to obtain a tongue contour parameterized curve of each frame image.

[0163] Specifically, the contour of the tongue can be fitted into a parameterized curve through a set of coordinate points (such as edge points), and the points on the curve are represented by parameter t. Then the parameterized curve C(t) can be expressed as:

[0164] C(t)=(x(t),y(t)),t∈[0,1]

[0165] Among them, x(t) and y(t) represent the coordinates of the tongue contour in the horizontal and vertical directions, respectively. The value range of parameter t is usually [0,1], which represents the normalized progress of the curve length.

[0166] According to the point set of the tongue contour, the parameterized curve of the tongue contour is reconstructed by spline interpolation method. The expression of the parameterized curve is:

[0167]

[0168] Where P represents the calculated tongue contour point set, N i (t) represents the spline basis function.

[0169] S304, analyzing the tongue contour parameterized curves of all frame images to obtain dynamic behavior data of the tongue contour.

[0170] Among them, the dynamic behavior data of the tongue contour in this embodiment can be studied by analyzing the changes in the time series C(t). The dynamic behavior data of the tongue contour, as a characteristic vector of a certain time step, can include the displacement, velocity, acceleration, tongue curvature, area change of the tongue area, etc.

[0171] S305, input the dynamic behavior data of the tongue contour into the neural network for the regression task, extract the deep features related to the vibration amplitude, frequency, duration, and vibration location, and record them as tongue vibration features.

[0172] Among them, the neural network used for the regression task can be a fully connected neural network, a recursive neural network, a long short-term memory network, etc., which is not limited in this embodiment.

[0173] It can be understood that this embodiment specifically proposes a quantitative feature extraction method for tongue tremor, an important indicator in traditional Chinese medicine tongue diagnosis. By using the Sobel operator for edge detection, the brightness change of the tongue area can be effectively identified, so as to accurately extract the tongue contour, track the coordinates of the tongue contour of each frame image, continuously record the changes in the tongue contour and fit the tongue contour point set into a parameterized curve, and obtain the dynamic behavior data of the tongue contour by analyzing the tongue contour parameterized curve. The neural network for regression tasks is used to extract the deep tongue tremor features related to the health status of the central nervous system, and they are used together with the static tongue features and the hyperspectral features in the subsequent analysis of the fused tongue image feature data, thereby achieving comprehensive consideration of the diagnostic information of the tongue health status, which helps to improve the fusion effect of the tongue features and the accuracy of the analysis results.

[0174] Specifically, in this embodiment, a deep neural network is used to extract features of the tongue region after registration in the hyperspectral image to obtain the tongue spectral features, including:

[0175] The aligned tongue area in the hyperspectral image is input into the three-dimensional convolutional neural network to extract the deep features related to the chemical composition of the tongue, which are recorded as the tongue spectral features.

[0176] It can be understood that since the hyperspectral image has a three-dimensional data structure (two spatial dimensions and one spectral dimension), the present embodiment uses a three-dimensional convolutional neural network (3D-CNN) to extract features of the tongue area, which can simultaneously process spatial and spectral information, thereby extracting features related to chemical composition.

[0177] Optionally, the tongue spectral features extracted in this embodiment include but are not limited to the following:

[0178] 1. Spectral reflectivity: Hyperspectral images record the reflectivity of objects at different wavelengths. These reflectivities are related to the chemical and physical properties of the objects. By analyzing the spectral reflectivity, the spectral characteristics of the tongue tissue can be obtained.

[0179] 2. Spectral slope: Spectral slope can help identify the shape changes of the spectral curve and reflect the changing trend of the spectral characteristics.

[0180] 3. Absorption characteristics: Different chemical substances will have specific absorption peaks at different wavelengths. By identifying these absorption peaks, the biochemical composition of tongue tissue can be inferred.

[0181] S105, fusing the tongue spectrum features, static tongue features and tongue tremor features to obtain fused tongue image feature data.

[0182] Optionally, the method for fusing tongue spectrum features, static tongue features, and tongue tremor features in this embodiment may adopt simple splicing, weighted summation, or other more complex fusion strategies, which is not limited in this embodiment.

[0183] S106, using a deep neural network to analyze the fused tongue image feature data and predict the TCM analysis result of the current tongue image.

[0184] Specifically, a deep neural network is used to analyze the fused tongue image feature data and predict the TCM analysis results of the current tongue image, including:

[0185] The fused tongue image feature data is input into the neural network for classification tasks to obtain the probability that the current tongue image belongs to different categories of TCM analysis results, and the category with the highest probability is selected as the prediction result of the current tongue image.

[0186] Optionally, the neural network used for the classification task in this embodiment may be a fully connected neural network, a convolutional neural network, a recurrent neural network, a deep belief network, etc., which is not limited in this embodiment.

[0187] For example, the fused tongue image feature data is input into the neural network for classification task to obtain the probability that the current tongue image belongs to different categories of TCM analysis results. The fused tongue image feature data is input into the neural network for classification task. Assume that the output is a vector z, where z∈R K , K is the number of categories of TCM analysis results. This vector z is the original prediction score of the TCM analysis results of each category by the neural network. Next, the softmax function is applied to the vector z so that the output can be interpreted as a probability distribution, converting the original prediction score into a probability value for each category, and the sum of all probability values ​​is 1.

[0188] Furthermore, after obtaining the predicted results of the TCM analysis of the current tongue image, the results can be fed back to the remote server, the multimodal data of the tongue surface can be displayed in real time on the display for the user to view, and a final analysis result report of each indicator of TCM tongue diagnosis can be provided.

[0189] In another embodiment of the present specification, compared with the aforementioned embodiment, this embodiment adds consideration of the visible light image of the tongue base when extracting static tongue features, wherein the implementation steps of amplifying the high-frequency components of the visible light image of the tongue base, extracting the tongue area, and image registration of the extracted tongue area are similar to the implementation steps of the visible light image in the tongue surface multimodal data in the aforementioned embodiment, and will not be repeated in this embodiment.

[0190] In this embodiment, the registered tongue region in the visible light image includes the tongue surface region image and the tongue base region image, and a deep neural network is used to extract features of the registered tongue region in the visible light image to obtain static tongue features, including:

[0191] The tongue surface area image and the tongue base area image are input into a two-branch convolutional neural network, and the two branches respectively extract features from the tongue surface area image and the tongue base area image, one branch extracts tongue surface depth features related to color, texture, shape, and local structure, and the other branch extracts tongue base depth features related to color, local structure, and tissue structure;

[0192] The tongue surface depth features and tongue base depth features are fused to obtain comprehensive tongue body features, which are recorded as static tongue features.

[0193] Among them, the convolutional neural network branch architecture of the tongue surface area image is similar to the convolutional neural network architecture for feature extraction of the tongue area after alignment in the visible light image in the aforementioned embodiment, and this embodiment will not be repeated here.

[0194] Since the features of the tongue base image may be more complex, including the tissue structure of the tongue base, possible lesion areas, etc., the tongue base image requires deeper contextual information to identify more complex local structures. Preferably, in this embodiment, for the tongue base area image, the convolutional neural network architecture of EfficientNet or UNet can be considered. The specific reasons are as follows:

[0195] On the one hand, as an efficient convolutional network, EfficientNet improves performance by scaling network width, depth, and resolution, and is suitable for efficient feature extraction under limited computing resources. For tongue base area images, EfficientNet can effectively capture global features in tongue base area images while retaining details through multi-scale feature fusion, which is suitable for the more complex structure of the tongue base area.

[0196] On the other hand, UNet, as a network architecture specifically used for image segmentation, has a symmetrical encoder-decoder structure. It can effectively extract local lesion features in the tongue base area image, while retaining multi-scale information through jump connections. It is particularly suitable for tasks that require extraction of tongue base area features, such as detection of lesion areas. Moreover, UNet can retain local details at different scales through its encoder-decoder structure, which is particularly suitable for areas with large local changes in tongue base area images.

[0197] It is understandable that there are significant visual differences between the visible light image of the tongue surface and the visible light image of the tongue base. For example, the tongue surface usually has more texture and local structural details, such as tongue coating, cracks, tongue veins, etc., while the local structure of the tongue base image may be more complex, including the tissue structure of the tongue base, possible lesion areas, etc. Therefore, this embodiment uses different convolutional neural network branch architectures for the tongue surface and tongue base images to extract their respective local features, which can better capture the local structure and texture information of the tongue surface and tongue base images, and provide richer feature representation for subsequent classification or regression tasks. At the same time, the type of lesions that may occur in the tongue base area is different from that in the tongue surface area. Therefore, the design of the dual-branch convolutional neural network in this embodiment can better adapt to these differences and improve the ability to predict or identify specific lesion types.

[0198] Please see attached Figure 2 , Figure 2 A schematic diagram of the structure of a tongue image feature analysis system provided in one embodiment of this specification.

[0199] like Figure 2 As shown, the tongue image feature analysis system may include at least an image acquisition module, an image processing module, a feature extraction module, a feature fusion module and a feature analysis module, wherein:

[0200] An image acquisition module is used to obtain multimodal image data of the tongue surface, wherein the multimodal image data of the tongue surface includes a hyperspectral image, a visible light image, and a dynamic tongue image;

[0201] An image processing module is used to amplify the high-frequency components of the visible light image and the dynamic tongue image to obtain the processed visible light image and the dynamic tongue image, extract the tongue area from the hyperspectral image, the processed visible light image and the dynamic tongue image, and perform image registration on the extracted tongue area;

[0202] A feature extraction module is used to extract features of the registered tongue area in the hyperspectral image, the visible light image and the dynamic tongue image using a deep neural network to obtain tongue spectral features, static tongue features and tongue tremor features;

[0203] A feature fusion module is used to fuse tongue spectral features, static tongue features and tongue tremor features to obtain fused tongue image feature data;

[0204] The feature analysis module is used to analyze the fused tongue image feature data using a deep neural network and predict the TCM analysis results of the current tongue image.

[0205] Among them, the image acquisition module is composed of three cameras, including a static image acquisition camera, a dynamic image acquisition camera and a multispectral camera. The static image acquisition camera and the dynamic image acquisition camera are fixed at preset positions, and the hyperspectral camera is fixed on an electrically controlled slide for scanning imaging to obtain hyperspectral image data.

[0206] For example, when a face is placed in the collection position, the system broadcasts a voice prompt, controls the static image collection camera to collect a face image, and controls the dynamic image collection camera to collect multiple tongue images in sequence at a fixed time interval t to form a dynamic tongue image sequence, and moves the slide to drive the hyperspectral camera to scan and image the tongue to obtain a hyperspectral image of the tongue.

[0207] It can be understood that the technical concept of a tongue image feature analysis system provided in this embodiment is similar to the technical concept of a tongue image feature analysis method provided in the aforementioned embodiment, and this embodiment will not be repeated here.

[0208] The above descriptions are only preferred embodiments disclosed in this application and descriptions of the technical principles used. Those skilled in the art should understand that the scope of protection involved in this disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in this disclosure (but not limited to) to form a technical solution.

[0209] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

Claims

1. A tongue image feature analysis method, characterized in that: The following steps are involved: Acquire tongue surface multimodal image data, wherein the tongue surface multimodal image data includes a hyperspectral image, a visible light image, and a dynamic tongue image; A high-frequency component amplification process is performed on the visible light image and the dynamic tongue image to obtain a processed visible light image and a dynamic tongue image; Extracting the tongue region from the hyperspectral image, the processed visible light image and the dynamic tongue image respectively, and performing image registration on the extracted tongue region; A deep neural network is used to extract features of the registered tongue regions in the hyperspectral image, visible light image and dynamic tongue image, respectively, to obtain tongue spectral features, static tongue features and tongue tremor features. The tongue spectrum features, static tongue features and tongue tremor features are fused to obtain fused tongue image feature data; A deep neural network is used to analyze the fused tongue image feature data and predict the TCM analysis results of the current tongue image.

2. A tongue image feature analysis method according to claim 1, characterized in that: The high-frequency component of the visible light image is amplified to obtain a processed visible light image, including: The visible light image is converted into the frequency domain using the discrete Fourier transform algorithm to obtain the frequency domain matrix of the visible light image; Creating a high-pass mask, and determining high-frequency components of the frequency domain matrix based on the high-pass mask; Adding a phase shift to the high frequency component to obtain a frequency domain matrix after adding the phase shift; The frequency domain matrix after adding the phase shift is inversely discrete Fourier transformed to obtain the processed visible light image.

3. A tongue image feature analysis method as claimed in claim 2, characterized in that: The "performing an inverse discrete Fourier transform on the frequency domain matrix after adding the phase shift to obtain a processed visible light image" includes: Calculate the phase gradient of each frequency component in the frequency domain matrix after adding the phase shift along the horizontal direction and the vertical direction; determining a phase change of each frequency component based on a phase gradient of each frequency component; Comparing the phase change of each frequency component with a preset phase change threshold, and determining adjacent frequency components whose phase change exceeds the preset phase change threshold based on the comparison result; constructing a weight matrix based on a set of adjacent frequency components whose phase changes exceed a preset phase change threshold; The weight matrix is ​​smoothed using a two-dimensional Gaussian function; Multiply the smoothed weight matrix by the frequency domain matrix after adding the phase shift to obtain the final frequency domain matrix; The final frequency domain matrix is ​​inversely discrete Fourier transformed to obtain the processed visible light image.

4. A tongue image feature analysis method according to claim 1, characterized in that: The tongue area is extracted from the hyperspectral image, the processed visible light image and the dynamic tongue image, including: a. Select hyperspectral image I D The image I of the Cth channel in D-C , according to the preset threshold, the image I of the Cth channel D-C The mask M corresponding to the tongue area is segmented out D ; The image I of the Cth channel D-C The mask M of the tongue area D Point-to-point multiplication is performed to obtain the tongue area in the hyperspectral image; b. The processed visible light image I V Convert from RGB space to YCbCr space and select image I of Cr channel V-Cr , according to the preset threshold from the image I of the Cr channel V-Cr The mask M corresponding to the tongue area is segmented out V ; The Cr channel image I V-Cr The mask M of the tongue area V Multiply point by point to obtain the tongue area in the visible light image; c. sequentially reading a number of frames of images from the processed dynamic tongue image to form a dynamic tongue image sequence; Each frame of the dynamic tongue image sequence is converted from RGB space to YCbCr space, and the image I of the Cr channel is selected. B-Cr , according to the preset threshold from the image I of the Cr channel B-Cr The mask M corresponding to the tongue area is segmented out B ; The Cr channel image I B-Cr The mask M of the tongue area B Multiply point by point to obtain the tongue area in the dynamic tongue image.

5. A tongue image feature analysis method as claimed in claim 4, characterized in that: A deep neural network is used to extract features of the registered tongue area in the visible light image to obtain static tongue features, including: The registered tongue area in the visible light image is input into the convolutional neural network to extract deep features related to color, texture, shape, and local structure, which are recorded as static tongue features.

6. A tongue image feature analysis method as claimed in claim 5, characterized in that: The registered tongue region in the visible light image includes a tongue surface region image and a tongue base region image. A deep neural network is used to extract features of the registered tongue region in the visible light image to obtain static tongue features, including: The tongue surface area image and the tongue base area image are input into a two-branch convolutional neural network, and the two branches respectively extract features from the tongue surface area image and the tongue base area image, one branch extracts tongue surface depth features related to color, texture, shape, and local structure, and the other branch extracts tongue base depth features related to color, local structure, and tissue structure; The tongue surface depth features and tongue base depth features are fused to obtain comprehensive tongue body features, which are recorded as static tongue features.

7. A tongue image feature analysis method as claimed in claim 4, characterized in that: A deep neural network is used to extract features from the registered tongue area in the dynamic tongue image to obtain tongue tremor features, including: The Sobel operator is used to perform edge detection on the registered tongue area in the dynamic tongue image to obtain the tongue contour of each frame in the dynamic tongue image sequence. Track the coordinates of the tongue contour of each frame image to obtain the tongue contour point set of each frame image; Fitting the tongue contour point set of each frame image to obtain the tongue contour parameterized curve of each frame image; Analyze the parameterized curves of tongue contours of all frame images to obtain the dynamic behavior data of tongue contours; The dynamic behavior data of the tongue contour is input into the neural network for regression tasks, and the deep features related to the vibration amplitude, frequency, duration, and vibration location are extracted and recorded as tongue vibration features.

8. A tongue image feature analysis method as claimed in claim 4, characterized in that: A deep neural network is used to extract features of the tongue area after registration in the hyperspectral image to obtain the tongue spectral features, including: The aligned tongue area in the hyperspectral image is input into the three-dimensional convolutional neural network to extract the deep features related to the chemical composition of the tongue, which are recorded as the tongue spectral features.

9. The tongue image feature analysis method according to claim 1, characterized in that: A deep neural network is used to analyze the fused tongue image feature data and predict the TCM analysis results of the current tongue image, including: The fused tongue image feature data is input into the neural network for classification tasks to obtain the probability that the current tongue image belongs to different categories of TCM analysis results, and the category with the highest probability is selected as the prediction result of the current tongue image.

10. A tongue image feature analysis system, characterized in that: include: An image acquisition module, used to acquire tongue surface multimodal image data, wherein the tongue surface multimodal image data includes a hyperspectral image, a visible light image, and a dynamic tongue image; An image processing module is used to amplify the high-frequency components of the visible light image and the dynamic tongue image to obtain the processed visible light image and the dynamic tongue image, extract the tongue area from the hyperspectral image, the processed visible light image and the dynamic tongue image, and perform image registration on the extracted tongue area; A feature extraction module is used to extract features of the registered tongue area in the hyperspectral image, the visible light image and the dynamic tongue image using a deep neural network to obtain tongue spectral features, static tongue features and tongue tremor features; A feature fusion module is used to fuse tongue spectral features, static tongue features and tongue tremor features to obtain fused tongue image feature data; The feature analysis module is used to analyze the fused tongue image feature data using a deep neural network and predict the TCM analysis results of the current tongue image.

Images