A tongue image feature analysis method and system

Abstract

The present application relates to the technical field of image processing, in particular to a tongue image feature analysis method and system. The method comprises: acquiring tongue surface multi-modal image data including hyperspectral images, visible light images and dynamic tongue images; performing high-frequency component amplification processing on the visible light images and the dynamic tongue images; extracting tongue region from the hyperspectral images, the processed visible light images and the dynamic tongue images respectively, and performing image registration on the extracted tongue region; using a deep neural network to extract features of the registered tongue region in the tongue surface multi-modal image data respectively, to obtain tongue spectral features, static tongue features and tongue tremor features, and to fuse them to obtain fused tongue image feature data; using a deep neural network to analyze the fused tongue image feature data and predict the traditional Chinese medicine analysis result of the current tongue image, reducing the dependence on complex network structure, and improving the fusion effect of tongue features and the accuracy of analysis results.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a tongue image feature analysis method and system. BACKGROUND

[0002] As a non-invasive examination method, tongue diagnosis has become an important part of traditional Chinese medicine diagnosis methods. Traditional Chinese medicine believes that the tongue is a mirror of the human body, and the tongue is closely related to the zang-fu organs and meridians, so the changes in the tongue can reveal the functional status of the internal zang-fu organs, and thus the health status of the human body can be reflected through the changes in the shape, color, and tongue fur area of the tongue.

[0003] With the application of big data technology and the progress of artificial intelligence algorithms, higher requirements are put forward for the scientificity and objectivity of traditional Chinese medicine diagnosis methods. Therefore, the objectification of tongue diagnosis has emerged as the times require. For example, Chinese patent 202410668227.7, a tongue image detection method based on RGB image and hyperspectral image fusion, extracts spatial features from RGB images by designing a space-graph subnetwork for enhancing the global features and local features of the perception image, designs a spectral subnetwork for capturing the long-range correlation of spectral features to extract spectral features from hyperspectral images, and then performs weighted summation on the spatial features and spectral features to obtain the final fusion probability, thereby processing the tongue image into class information, which contains more biological information and is conducive to the accurate classification of health status. However, the existing patent still has the following technical problems:

[0004] 1. In the frequency domain of natural images, the energy distribution is mainly concentrated in the low-frequency region, and the high-frequency components are relatively few. However, the sharp transitions and texture features of the tongue structure in the tongue image mainly exist in the high-frequency part. In order to enhance the global features and local features of the perception RGB image and the spectral features, the patent designs a complex space-graph subnetwork and a spectral subnetwork, which contains multiple attention dense blocks and transition layers, resulting in a large amount of calculation in the network training process and easy overfitting. The complex network structure also makes the model more sensitive to external factors such as tongue image noise and light changes, resulting in insufficient robustness of the model;

[0005] 2. Due to differences in acquisition equipment and conditions, the resolution, viewing angle, and lighting conditions of the obtained RGB images and hyperspectral images will inevitably differ, which can easily produce false local feature information or introduce artifacts in the fusion process, thereby adversely affecting the accuracy of the feature fusion effect and analysis results;

[0006] 3. The patent only considers the spatial features and spectral features of static images, ignoring the evaluation indicators related to the health status of the central nervous system, resulting in incomplete diagnosis information for the health status of the tongue and limiting the in-depth analysis of the disease mechanism by the neural network. SUMMARY

[0007] In order to solve the above technical problems, the present application provides a tongue image feature analysis method and system, which aims to optimize the design of image processing algorithm, to strengthen the key details in the image, to reduce the complexity of the network structure, to combine the comprehensive consideration of the diagnosis information of the tongue health status, and to further improve the fusion effect of the tongue features and the accuracy of the analysis results.

[0008] In a first aspect, the present application provides a tongue image feature analysis method, comprising the following steps:

[0009] Obtaining tongue surface multi-modal image data, the tongue surface multi-modal image data comprising hyperspectral images, visible light images, and dynamic tongue images;

[0010] Performing high-frequency component amplification processing on the visible light images and the dynamic tongue images to obtain processed visible light images and dynamic tongue images;

[0011] Extracting tongue regions from the hyperspectral images, the processed visible light images, and the dynamic tongue images respectively, and performing image registration on the extracted tongue regions;

[0012] Using a deep neural network to extract features from the registered tongue regions in the hyperspectral images, the visible light images, and the dynamic tongue images respectively to obtain tongue spectral features, static tongue features, and tongue tremor features;

[0013] Fusing the tongue spectral features, the static tongue features, and the tongue tremor features to obtain fused tongue image feature data;

[0014] Using a deep neural network to analyze the fused tongue image feature data and predicting a traditional Chinese medicine analysis result of the current tongue image.

[0015] In some embodiments, the high-frequency component amplification processing on the visible light images to obtain the processed visible light images comprises:

[0016] Using a discrete Fourier transform algorithm to convert the visible light images to the frequency domain to obtain a frequency domain matrix of the visible light images;

[0017] Creating a high-pass mask and determining high-frequency components of the frequency domain matrix based on the high-pass mask;

[0018] Adding a phase shift to the high-frequency components to obtain a frequency domain matrix after adding the phase shift;

[0019] Performing inverse discrete Fourier transform on the frequency domain matrix after adding the phase shift to obtain the processed visible light images.

[0020] In some embodiments, the "performing inverse discrete Fourier transform on the frequency domain matrix after adding the phase shift to obtain the processed visible light image" comprises:

[0021] calculating phase gradients of each frequency component in the frequency domain matrix after adding the phase shift along the horizontal direction and the vertical direction;

[0022] determining phase changes of each frequency component based on the phase gradients of each frequency component;

[0023] comparing the phase changes of each frequency component with a preset phase change threshold, and determining adjacent frequency components whose phase changes exceed the preset phase change threshold based on a comparison result;

[0024] constructing a weight matrix based on a set of the adjacent frequency components whose phase changes exceed the preset phase change threshold;

[0025] performing smoothing processing on the weight matrix by using a two-dimensional Gaussian function;

[0026] multiplying the smoothed weight matrix with the frequency domain matrix after adding the phase shift to obtain a final frequency domain matrix;

[0027] performing inverse discrete Fourier transform on the final frequency domain matrix to obtain the processed visible light image.

[0028] In some embodiments, the tongue region is extracted from the hyperspectral image, the processed visible light image and the dynamic tongue image respectively, comprising:

[0029] a. selecting an image I D of the Cth channel in the hyperspectral image I D-C , and segmenting a mask M D corresponding to the tongue region from the image I D-C of the Cth channel according to a preset threshold;

[0030] point-by-point multiplying the image I D-C of the Cth channel with the mask M D of the tongue region to obtain the tongue region in the hyperspectral image;

[0031] b. converting the processed visible light image I V from the RGB space to the YCbCr space, selecting an image I V-Cr of the Cr channel, and segmenting a mask M V corresponding to the tongue region from the image I V-Cr of the Cr channel according to a preset threshold;

[0032] point-by-point multiplying the image I V-Cr of the Cr channel with the mask M V of the tongue region to obtain the tongue region in the visible light image;

[0033] c. reading a plurality of frames from the processed dynamic tongue image in sequence to form a dynamic tongue image sequence;

[0034] converting each frame in the dynamic tongue image sequence from RGB space to YCbCr space, selecting the image I B-Cr from the Cr channel according to a preset threshold to segment a mask M B-Cr corresponding to the tongue region from the image I B ;

[0035] point-by-point multiplying the image I B-Cr from the Cr channel and the mask M B of the tongue region to obtain the tongue region in the dynamic tongue image.

[0036] In some embodiments, a deep neural network is used to extract features from the registered tongue region in the visible light image to obtain static tongue features, including:

[0037] The registered tongue region in the visible light image is input into a convolutional neural network to extract deep features related to color, texture, shape, and local structure, denoted as static tongue features.

[0038] In some embodiments, the registered tongue region in the visible light image includes a tongue surface region image and a tongue base region image, and a deep neural network is used to extract features from the registered tongue region in the visible light image to obtain static tongue features, including:

[0039] The tongue surface region image and the tongue base region image are input into a dual-branch convolutional neural network, and the two branches extract features from the tongue surface region image and the tongue base region image respectively. One branch extracts tongue surface deep features related to color, texture, shape, and local structure, and the other branch extracts tongue base deep features related to color, local structure, and tissue structure.

[0040] The tongue surface deep features and the tongue base deep features are fused to obtain comprehensive tongue features, denoted as static tongue features.

[0041] In some embodiments, a deep neural network is used to extract features from the registered tongue region in the dynamic tongue image to obtain tongue tremor features, including:

[0042] A Sobel operator is used to perform edge detection on the registered tongue region in the dynamic tongue image to obtain tongue contours for each frame in the dynamic tongue image sequence.

[0043] Coordinate tracking is performed on the tongue contours of each frame to obtain a set of tongue contour points for each frame.

[0044] fitting the tongue contour point set of each frame of image to obtain a parameterized curve of the tongue contour of each frame of image;

[0045] analyzing the parameterized curves of the tongue contour of all frames of image to obtain dynamic behavior data of the tongue contour;

[0046] inputting the dynamic behavior data of the tongue contour into a neural network for a regression task to extract deep features related to the tremor amplitude, frequency, duration and tremor site, denoted as tongue tremor features.

[0047] In some embodiments, a deep neural network is used to extract features from the registered tongue region in the hyperspectral image to obtain tongue spectral features, including:

[0048] The registered tongue region in the hyperspectral image is input into a three-dimensional convolutional neural network to extract deep features related to the chemical composition of the tongue, denoted as tongue spectral features.

[0049] In some embodiments, a deep neural network is used to analyze the fused tongue feature data and predict the traditional Chinese medicine analysis result of the current tongue image, including:

[0050] The fused tongue feature data is input into a neural network for a classification task to obtain the probability of the current tongue image belonging to different categories of traditional Chinese medicine analysis results, and the category with the highest probability is selected as the predicted result of the current tongue image.

[0051] In a second aspect, the present application provides a tongue feature analysis system, including:

[0052] An image acquisition module is configured to acquire tongue surface multi-modal image data, the tongue surface multi-modal image data including a hyperspectral image, a visible light image and a dynamic tongue image;

[0053] An image processing module is configured to perform high-frequency component amplification processing on the visible light image and the dynamic tongue image to obtain processed visible light image and dynamic tongue image, extract tongue regions from the hyperspectral image, the processed visible light image and the dynamic tongue image respectively, and perform image registration on the extracted tongue regions;

[0054] A feature extraction module is configured to use a deep neural network to extract features from the registered tongue regions in the hyperspectral image, the visible light image and the dynamic tongue image respectively to obtain tongue spectral features, static tongue features and tongue tremor features;

[0055] A feature fusion module is configured to fuse the tongue spectral features, the static tongue features and the tongue tremor features to obtain fused tongue feature data;

[0056] The feature analysis module is configured to analyze the fused tongue feature data by using a deep neural network, and predict a traditional Chinese medicine analysis result of the current tongue image.

[0057] The present application has at least the following beneficial technical effects:

[0058] 1. The tongue feature analysis method and system, through the optimization design of the multi-modal image processing algorithm of the tongue surface, only adds a phase shift to the high-frequency component in the frequency domain, and through the phase adjustment of the high-frequency part, the key detail features in the multi-modal image data of the tongue surface are greatly strengthened, and the visual effect of the visible light image is improved, so that even if the general deep neural network is used, the pathological features of the tongue image can still be learned and recognized more accurately, thereby improving the classification or recognition accuracy of the final traditional Chinese medicine analysis result, reducing the dependence on the complex network structure, reducing the calculation amount of the deep neural network, making the neural network more stable and reliable in the actual tongue feature analysis, and considering the tongue tremor feature which is an important index related to the health status of the central nervous system while extracting the static tongue feature and the tongue spectrum feature, realizing the comprehensive consideration of the diagnosis information of the tongue health status, and helping to improve the fusion effect of the tongue feature and the accuracy of the analysis result.

[0059] 2. The visible light image is converted to the frequency domain by using the discrete Fourier transform algorithm, only the high-frequency component is added with a phase shift in the frequency domain, which is equivalent to changing the phase of these high-frequency components in the spatial domain, and through the phase adjustment of the high-frequency part, the enhancement of the tongue image details (such as edges and textures) is realized, so that the visible light image is clearer, and the visual effect of the visible light image is improved. Further, in order to highlight the high-frequency part, a weighting scheme based on the phase change between adjacent frequency domain components is proposed, and a two-dimensional Gaussian function is used to smooth the weight matrix, so as to avoid the phenomenon that the tongue image appears artifacts due to the sharp change of the weight matrix, so that the smoothed weight matrix can effectively amplify the high-frequency component to enhance the key details and edge information suppressed in the high-frequency component. The final frequency domain matrix after the weighting processing based on the phase change between adjacent frequency domain components is inversely discrete Fourier transformed, and the image after modifying the phase is converted back to the spatial domain. The processed visible light image obtained in this way contains further highlighted high-frequency components, thereby further improving the visual effect of the visible light image, and then performing image registration after extracting the tongue region in the three images, which helps to obtain more accurate feature vectors in the feature extraction stage, thereby greatly reducing the adverse effects on the subsequent feature fusion effect and the accuracy of the analysis result.

[0060] 3、Generally, the visible light image is divided into three channels of RGB, but the difference between the tongue region and the background region in the visible light image of the tongue is not obvious in the RGB three channels, and the image in the RGB space is greatly affected by the light, therefore, the visible light image and the tongue dynamic image are converted from the RGB space to the YCbCr space, so that the difference between the tongue region and the background region is significantly displayed, and the image in the Cr channel (i.e. the red channel) in the YCbCr space is selected as the basis for extracting the tongue region, because the color of the tongue is usually reddish, therefore, in the Cr channel, the contrast between the tongue region and the background region is often higher, so that the tongue region is more prominent, and the Cr channel has stronger adaptability to different light conditions and background colors, so that the tongue region can be better extracted in different environments, which helps to improve the accuracy of extracting the tongue region in the visible light image and the tongue dynamic image, and reduces the cases of false segmentation and missed segmentation;

[0061] 3、For the tongue tremor feature, which is an important index related to the health status of the central nervous system, a targeted quantitative feature extraction method is designed, because the dynamic tongue image is continuously changing, the position and shape of the tongue contour in each frame of image may be different, in order to better track the change of the tongue contour, the Sobel operator is used for edge detection to expand the image gradient of the tongue region in the horizontal and vertical directions, and the change intensity of the tongue contour in the two directions is obtained, after adding the gradient information, the moving path of the tongue contour can be more accurately predicted and tracked, so that the tongue contour can be accurately extracted, the tongue contour line extracted from each frame of image is tracked in coordinates, the change of the tongue contour is continuously recorded and the tongue contour point set is fitted into a parameterized curve, the dynamic behavior data of the tongue contour is studied by analyzing the change in the time sequence parameterized curve, the deep tongue tremor features related to the tremor amplitude, frequency, duration and tremor site are extracted by using the neural network for regression task, and are used together with the static tongue features and the hyperspectral features for subsequent analysis of the fused tongue image feature data, so that the diagnosis information of the tongue health status is comprehensively considered, which helps to improve the fusion effect of the tongue features and the accuracy of the analysis result;

[0062] 4、Since the visible light image of the tongue surface and the visible light image of the tongue bottom have significant differences in vision, for example, the tongue surface usually has more texture and local structure details such as tongue fur, cracks, tongue veins, etc., and the local structure of the tongue bottom image can be more complex, including the tissue structure of the tongue bottom, possible lesion areas, etc., therefore, the present application adopts different convolutional neural network branch architectures for the tongue surface and tongue bottom images respectively to extract their respective local features, which can better capture the local structure and texture information of the tongue surface and tongue bottom images, and provide more rich feature representations for subsequent classification or regression tasks, at the same time, the lesion types that can appear in the tongue bottom region are different from those in the tongue surface region, therefore, the design of the double-branch convolutional neural network in the present application can better adapt to these differences and improve the prediction or recognition ability of specific lesion types.

[0063] Other features and advantages of the present application will be disclosed in detail in the following specific embodiments and drawings. BRIEF DESCRIPTION OF DRAWINGS

[0064] The present application will be further described below in conjunction with the drawings:

[0065] Figure 1 The tongue image feature analysis method flowchart of the embodiment of the present application.

[0066] Figure 2 The structure schematic diagram of the tongue image feature analysis system of the embodiment of the present application. DETAILED DESCRIPTION

[0067] The technical solutions of the embodiments of the present application will be explained and described below in conjunction with the drawings of the embodiments of the present application, but the following embodiments are only preferred embodiments of the present application, not all. Based on the embodiments in the embodiments, other embodiments obtained by those skilled in the art without creative labor are all within the protection scope of the present application.

[0068] In the following description, the appearance of terms such as "inner", "outer", "upper", "lower", "left", "right", etc. only indicates the orientation or positional relationship for the convenience of describing the embodiments and simplifying the description, and does not indicate or imply that the indicated device or element must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the present application.

[0069] Please refer to the drawings Figure 1 , Figure 1 The flowchart of the tongue image feature analysis method provided by one embodiment of the present application is shown.

[0070] As Figure 1 shown, the tongue image feature analysis method can at least include the following steps:

[0071] S101, obtain tongue surface multi-modal image data, the tongue surface multi-modal image data including a hyperspectral image, a visible light image and a dynamic tongue image.

[0072] The hyperspectral image is an image capable of capturing the reflection spectrum information of the face and tongue in multiple continuous spectral bands (usually from ultraviolet to near infrared range), and is a three-dimensional data structure, usually including two-dimensional spatial coordinates (or pixel positions) and wavelengths, and the wavelength corresponds to the spectral intensity value of each pixel point, usually in nanometers (nm).

[0073] The dynamic tongue image can be a dynamic tongue video taken, or can be composed of multiple tongue images sequentially collected by a dynamic image collection camera at a fixed time interval t.

[0074] Further, in order to improve the effectiveness of the obtained tongue surface multi-modal image data, the method for obtaining the multi-modal image data of the tongue surface can further include the following steps:

[0075] Real-time acquisition of a human face image;

[0076] Using a face key point detection algorithm, the key point coordinates around the mouth in the human face image are obtained;

[0077] The distance between the key point coordinates is calculated and compared with a preset threshold value, if the distance exceeds the preset threshold value, it is determined that the mouth is open, and the next step is executed:

[0078] In the image in which the mouth is open, the threshold segmentation is performed on the internal region of the mouth to extract the possible tongue region;

[0079] The area of the tongue region or the number of pixels of the tongue region is calculated, and compared with a preset threshold value, if the calculation result exceeds the preset threshold value, it is determined that the tongue is extended;

[0080] When the determination result satisfies that the mouth is open and the tongue is extended, the image collection module is triggered to obtain the required hyperspectral image, visible light image and dynamic tongue image to constitute the tongue surface multi-modal image data.

[0081] By detecting the face key points in real time and determining the open state of the mouth and the extended state of the tongue based on the preset threshold value, the collection of the tongue surface multi-modal image data is triggered, which can effectively improve the effectiveness of the collection of the tongue surface multi-modal image data, thereby improving the overall tongue image feature analysis efficiency.

[0082] S102, performing high-frequency component amplification processing on the visible light image and the dynamic tongue image to obtain the processed visible light image and the processed dynamic tongue image.

[0083] In the frequency domain of natural images, the energy distribution is mainly concentrated in the low frequency area, and the high frequency component is relatively small, while the sharp transition and texture features of the tongue structure in the tongue image mainly exist in the high frequency part. Therefore, the embodiment proposes an optimization algorithm for tongue surface multi-modal image processing, which aims to effectively enhance the key details in the tongue image.

[0084] Specifically, in the embodiment, the visible light image is subjected to high frequency component amplification processing to obtain a processed visible light image, which includes:

[0085] S201, using a discrete Fourier transform algorithm to convert the visible light image to the frequency domain to obtain a frequency domain matrix of the visible light image.

[0086] Let the original visible light image be x[m,n], first use the discrete Fourier transform (Discrete Fourier Transform, DFT) algorithm to convert the original visible light image from the spatial domain to the frequency domain to obtain the frequency domain matrix X[u,v] of the visible light image as shown in the following formula (0.1), and the size of the frequency domain matrix is

[0087]

[0088] Where |X[u,v]| is the modulus of X[u,v], that is, the amplitude spectrum of the frequency component, which represents the signal strength or energy size at position (u,v) in the frequency domain matrix. jφ[u,v] is the phase spectrum of X[u,v], where φ[u,v] is the phase angle, which determines the phase information of each frequency component.

[0089] S202, create a high-pass mask, and determine the high frequency component of the frequency domain matrix based on the high-pass mask.

[0090] For example, the expression of the high-pass mask in the embodiment is shown in the following formula (0.2):

[0091]

[0092] Where u c and v c are preset cutoff frequencies. Then, the high-pass mask is multiplied by the frequency domain matrix, so that only the high frequency component is retained, and the low frequency component is weakened or eliminated.

[0093] S203, adding a phase shift to the high frequency component to obtain an added phase shift frequency domain matrix.

[0094] It can be understood that the low-frequency components in the frequency domain correspond to the smooth regions in the visible light image, and adding a phase shift θ to these low-frequency components can cause artifacts in the smooth regions of the image, thereby reducing the quality of the visible light image. To this end, the embodiment only adds a phase shift to the high-frequency components in the frequency domain, which is equivalent to changing the phase of these high-frequency components in the spatial domain. Through the phase adjustment of the high-frequency part, the enhancement of the details (such as edges and textures) of the tongue image is realized, the visible light image is made clearer, and the visual effect of the visible light image is improved.

[0095] After the operation of adding a phase shift θ, the frequency spectrum of the visible light image becomes as shown in the following formula (0.3):

[0096] X p [u,v]=|X[u,v]|e j(φ[u,v]+H[u,v]θsgn(φ[u,v])) (0.3)

[0097] Wherein, sgn is an operator, used to maintain the conjugate symmetry of the discrete Fourier transform.

[0098] S204, inverse discrete Fourier transform is performed on the frequency domain matrix after adding the phase shift to obtain a processed visible light image.

[0099] It can be understood that the inverse discrete Fourier transform (IDFT) is performed on the frequency domain matrix after adding the phase shift, which is converted back to the spatial domain. The processed visible light image obtained in this way contains amplified high-frequency components, because the influence of low-frequency components has been weakened, and high-frequency details have been preserved or even enhanced, thereby effectively enhancing the image contrast and enhancing the key details and edge information in the image.

[0100] It can be understood that the method for amplifying high-frequency components of the dynamic tongue image in the embodiment is: reading the dynamic tongue image frame by frame in order to obtain a dynamic tongue image sequence, and then performing the same high-frequency component amplification processing steps S201-S204 on the dynamic tongue image sequence as the visible light image to obtain a processed dynamic tongue image. The embodiment will not be repeated here.

[0101] Further, in order to highlight the high-frequency part, the embodiment proposes a weighting scheme based on the phase change between adjacent frequency domain components. Specifically, in the embodiment, "performing inverse discrete Fourier transform on the frequency domain matrix after adding the phase shift to obtain a processed visible light image" includes:

[0102] S2041, calculating the phase gradient of each frequency component in the frequency domain matrix after adding the phase shift along the horizontal direction and the vertical direction.

[0103] It can be understood that the embodiment obtains the phase change by calculating the phase gradient of each frequency component in the frequency domain matrix along the horizontal direction and the vertical direction. Wherein, the phase gradient is calculated as shown in the following formula (0.4):

[0104]

[0105] S2042, based on the phase gradient of each frequency component, determine the phase change of each frequency component.

[0106] For the phase gradient of each frequency component, the amplitude of the phase gradient is calculated, which represents the size of the phase change, that is, the phase change of each frequency component.

[0107] S2043, compare the phase change of each frequency component with the preset phase change threshold, and determine the adjacent frequency components whose phase change exceeds the preset phase change threshold based on the comparison result.

[0108] Wherein, the preset phase change threshold in the embodiment is used to distinguish the significant phase change and the insignificant phase change, and the preset phase change threshold can be fixed or adaptive threshold based on the content of the tongue image, which is not limited in the embodiment.

[0109] For example, the implementation of comparing the phase change of each frequency component with the preset phase change threshold and determining the adjacent frequency components whose phase change exceeds the preset phase change threshold based on the comparison result is as follows: traverse each frequency component in the frequency domain matrix, compare the phase change of each frequency component with the preset phase change threshold, if the phase change of a frequency component is greater than the preset phase change threshold, check the phase change of the adjacent frequency components (frequency components in the horizontal direction and the vertical direction) of the frequency component, if the phase change of the adjacent frequency components is also greater than the preset phase change threshold, then the two frequency components are adjacent frequency components with significant phase change.

[0110] S2044, based on the set of adjacent frequency components whose phase change exceeds the preset phase change threshold, construct a weight matrix.

[0111] For example, based on the weight matrix W p (u,v) of the phase change, as shown in the following formula (2.5):

[0112]

[0113] Wherein, α is a preset proportion factor, and S represents the set of adjacent frequency components whose phase change exceeds the preset phase change threshold τ φ .

[0114] S2045, smooth the weight matrix by using a two-dimensional Gaussian function.

[0115] The embodiment uses a two-dimensional Gaussian function g[u, v] to smooth the weight matrix W P [u, v], so as to avoid the phenomenon that the tongue image appears artifacts due to the sharp change of the weight matrix W P [u, v]. Let the smoothed weight matrix be W f [u, v], which is defined as shown in the following formula (2.6):

[0116]

[0117] Wherein, “*” is a convolution operator.

[0118] S2046, multiply the smoothed weight matrix and the frequency domain matrix after adding the phase shift to obtain the final frequency domain matrix.

[0119] It can be understood that the smoothed weight matrix can effectively amplify the high-frequency components to enhance the key details and edge information suppressed in the high-frequency components.

[0120] S2047, perform inverse discrete Fourier transform on the final frequency domain matrix to obtain the processed visible light image.

[0121] It can be understood that the inverse discrete Fourier transform is performed on the final frequency domain matrix after the weighting processing based on the phase change between adjacent frequency domain components, the image after modifying the phase is converted back to the spatial domain, and the processed visible light image obtained in this way contains further highlighted high-frequency components, thereby further improving the visual effect of the visible light image.

[0122] S103, respectively extract the tongue region from the hyperspectral image, the processed visible light image and the dynamic tongue image, and perform image registration on the extracted tongue region.

[0123] Specifically, in the embodiment, the tongue region is extracted from the hyperspectral image, the processed visible light image and the dynamic tongue image, including:

[0124] a. Select the image I D C in the hyperspectral image I D-C , and segment the mask M D corresponding to the tongue region from the image I D-C C according to a preset threshold;

[0125] Point-to-point multiplication of the image I D-C C and the mask M D of the tongue region to obtain the tongue region in the hyperspectral image.

[0126] It is understandable that in hyperspectral images, different channels represent different wavelengths, and each channel can provide different information about the hyperspectral image of the tongue surface. In this embodiment, the selection of the Cth channel for the hyperspectral image of the tongue surface depends on the following factors:

[0127] 1. Contrast: Select a channel with high contrast between the tongue area and the background (such as other parts of the mouth, teeth, etc.), which means that the reflection or emission characteristics of the tongue area are significantly different from other areas in this channel.

[0128] 2. Tongue features: An ideal passage may better highlight specific features of the tongue surface, such as blood vessels, tongue coating, and color changes.

[0129] 3. Noise level: Selecting a channel with lower noise helps improve the accuracy of threshold segmentation.

[0130] Preferably, in this embodiment, the near-infrared channel of the hyperspectral image is selected for tongue region extraction because, under normal circumstances, the near-infrared channel can penetrate the skin better and highlight features such as blood vessels, making it more suitable for tongue region extraction.

[0131] b. The processed visible light image I V Image I, converted from RGB space to YCbCr space, selecting the Cr channel. V-Cr Based on a preset threshold, the image I from the Cr channel... V-Cr The mask M is used to segment the corresponding tongue region. V ;

[0132] Image I of the Cr channel V-Cr Mask M of the tongue area V Point-to-point multiplication yields the tongue region in the visible light image.

[0133] Understandably, visible light images are typically divided into three channels: RGB. However, in visible light images of the tongue, the difference between the tongue region and the background region is not significant in the RGB channels. Moreover, images in RGB space are greatly affected by lighting conditions. Therefore, this embodiment converts the visible light image from RGB space to YCbCr space, making the difference between the tongue region and the background region more significant. The image of the Cr channel (i.e., the red channel) in YCbCr space is selected as the basis for tongue region extraction. Since the color of the tongue is usually reddish, the contrast between the tongue region and the background region is often higher in the Cr channel, making the tongue region more prominent. Furthermore, the Cr channel is more adaptable to different lighting conditions and background colors, thus it can extract the tongue region better in different environments. This helps to improve the accuracy of tongue region extraction in visible light images and reduce missegmentation and missed segmentation.

[0134] c. reading a plurality of frames of images in sequence from the processed dynamic tongue image to form a dynamic tongue image sequence;

[0135] Converting each frame of image in the dynamic tongue image sequence from RGB space to YCbCr space, selecting the image I B-Cr from the Cr channel, and segmenting a mask M B-Cr corresponding to the tongue region from the image I B according to a preset threshold value;

[0136] Point-to-point multiplying the image I B-Cr from the Cr channel and the mask M B of the tongue region to obtain the tongue region in the dynamic tongue image.

[0137] Using the same tongue region extraction method for the dynamic tongue image as for the visible light image also helps to improve the accuracy of tongue region extraction for each frame of image in the dynamic tongue image sequence, and reduce the cases of false segmentation and missed segmentation.

[0138] Further, in order to extract and express the image components most related to the tongue features from the hyperspectral image, the processed visible light image and the dynamic tongue image, so that the subsequent feature extraction work can grasp the most essential shape features of the tongue, the mask of the tongue region can also be morphologically processed, and the largest closed loop region is taken as the tongue region, so as to eliminate the noise caused by skin patches, and an expansion operation is further used to optimize the mask morphology of the tongue region.

[0139] For example, in the embodiment, the implementation manner of image registration for the extracted tongue region can be:

[0140] a. scaling M B and M D to obtain the scaled M and M , wherein β B and β D are scaling coefficients obtained by calibration in advance;

[0141] b. registering M and M V , considering translation transformation, and obtaining the deviation values Δx D and Δy D in two directions in the translation transformation by traversal;

[0142] c. calculating the deviation values Δx D and Δy D in the same manner as b. using M and M V .D );

[0143] d. According to the pre-calibrated scaling coefficient and the calculated deviation value, corresponding scaling and translation processing is performed on the tongue region in the dynamic tongue image and the tongue region in the hyperspectral image, and finally the registered tongue region is obtained.

[0144] In S104, a deep neural network is used to extract features from the registered tongue region in the hyperspectral image, the visible light image and the dynamic tongue image, respectively, to obtain tongue spectral features, static tongue features and tongue tremor features.

[0145] It can be understood that in the embodiment, the optimization design of the image processing algorithm through the above steps S101-S103 has greatly strengthened the key detail features in the tongue multi-modal image data, so that even if a general deep neural network is used in the embodiment, the pathological features of the tongue image can still be more accurately learned and recognized, thereby improving the classification or recognition accuracy of the final Chinese medicine analysis result, reducing the dependence on complex network structure, reducing the calculation amount of the deep neural network, making the neural network more stable and reliable in the actual tongue feature analysis, considering the tongue tremor feature which is an important indicator related to the health status of the central nervous system while extracting the static tongue features and the tongue spectral features, realizing comprehensive consideration of the diagnosis information of the tongue health status, and helping to improve the fusion effect of the tongue features and the accuracy of the analysis result.

[0146] Specifically, in the embodiment, a deep neural network is used to extract features from the registered tongue region in the visible light image to obtain static tongue features, including:

[0147] The registered tongue region in the visible light image is input into a convolutional neural network to extract deep features related to color, texture, shape and local structure, denoted as static tongue features.

[0148] Preferably, in the embodiment, for the visible light image of the tongue surface, a convolutional neural network architecture of ResNet (Residual Network) or DenseNet (Densely Connected Network) can be considered. The specific reasons are as follows:

[0149] On the one hand, ResNet can effectively solve the gradient vanishing problem of deep network, and its residual module helps the network to maintain high feature transmission capability when extracting image details in deep layers, effectively improves the training speed, and is suitable for extracting texture, cracks and other detail information in the tongue surface image. By increasing the depth of the network, ResNet can capture higher level image information;

[0150] On the other hand, DenseNet can effectively preserve multi-layer information and ensure efficient feature transmission through dense connection of each layer and all previous layers, and is particularly suitable for fine tongue texture feature extraction, which helps to improve the expression ability of static tongue features and the stability of the network.

[0151] In the embodiment, the convolutional neural network extracts deep features related to color, texture, shape, and local structure, and the specific implementation is as follows:

[0152] 1. Color feature: Since the image of the Cr channel is selected in the embodiment, the feature related to the color information, especially the information related to red, is focused on, so the convolutional neural network can more easily capture the color distribution and changes of the tongue region.

[0153] 2. Texture feature: The convolutional neural network can effectively extract the texture information of the tongue region, such as the roughness of the tongue coating, the shape and direction of the cracks, etc.

[0154] 3. Shape feature: Through the combination of convolutional layers and pooling layers, the convolutional neural network can recognize the contour, edge, and shape features of the tongue, such as the overall shape of the tongue, the contour of the tongue boundary, etc.

[0155] 4. Local structure feature: The convolutional neural network can learn the local structure features such as the distribution of the tongue veins, the local pattern of the tongue coating, and the specific shape of the cracks, and capture the spatial relationship between the tongue features, such as the relative position of the tongue veins and the tongue coating, the distribution of the cracks, etc.

[0156] Specifically, in the embodiment, a deep neural network is used to extract features from the registered tongue region in the dynamic tongue image, and the tongue tremor features are obtained, including:

[0157] S301, using the Sobel operator to perform edge detection on the registered tongue region in the dynamic tongue image, to obtain the tongue contour of each image in the dynamic tongue image sequence.

[0158] The purpose of edge detection is to identify points with obvious brightness changes in the tongue region, which form the contour of the tongue.

[0159] Since the dynamic tongue image is continuously changing, the position and shape of the tongue contour in each frame of image may be different, in order to better track the changes of the tongue contour, the Sobel operator is used in the embodiment to expand the image gradient of the tongue region from the horizontal and vertical directions, to obtain the change intensity of the tongue contour in the two directions. After adding the gradient information, the moving path of the tongue contour can be more accurately predicted and tracked.

[0160] S302, coordinate tracking is performed on the tongue contour of each image to obtain the tongue contour point set of each image.

[0161] Specifically, the tongue contour line extracted for each frame of image is tracked in coordinates, and the coordinate positions of each point on the contour are recorded. The set of coordinate points constitutes a set of tongue contour points, representing the accurate position of the tongue contour. The tracked set of contour points can be saved in the form of a list, an array, or other data structures for subsequent shape analysis, geometric measurement, or other forms of analysis.

[0162] S303, fitting the set of tongue contour points of each frame of image to obtain the parameterized curve of the tongue contour of each frame of image.

[0163] Specifically, the contour of the tongue part can be fitted into a parameterized curve by a set of coordinate points (such as edge points). If a point on the curve is represented by a parameter t, the parameterized curve C(t) can be expressed as:

[0164] C(t) = (x(t), y(t)), t ∈ [0, 1]

[0165] Where x(t) and y(t) represent the coordinates of the tongue contour in the horizontal and vertical directions respectively, and the value range of the parameter t is usually [0, 1], representing the normalized progress of the curve length.

[0166] According to the set of points of the tongue contour, the parameterized curve of the tongue contour is reconstructed by spline interpolation, and the expression of the parameterized curve is:

[0167]

[0168] Where P represents the set of tongue contour points calculated, N i (t) represents the spline basis function.

[0169] S304, analyzing the parameterized curve of the tongue contour of all frames of image to obtain the dynamic behavior data of the tongue contour.

[0170] Where the dynamic behavior data of the tongue contour in the embodiment can be studied by analyzing the changes in the time series C(t). The dynamic behavior data of the tongue contour as a feature vector of a certain time step can include the displacement, speed, acceleration, tongue curvature, and area change of the tongue region of the tongue contour.

[0171] S305, inputting the dynamic behavior data of the tongue contour into a neural network for regression task to extract deep features related to tremor amplitude, frequency, duration, and tremor site, denoted as tongue tremor features.

[0172] Where the neural network for regression task can be a fully connected neural network, a recurrent neural network, a long short-term memory network, etc. The embodiment does not limit this.

[0173] It can be understood that the embodiment proposes a targeted quantitative feature extraction method for the important index of tongue tremor in traditional Chinese medicine tongue diagnosis. By using the Sobel operator for edge detection, the brightness change of the tongue region can be effectively identified, so as to accurately extract the tongue contour. The coordinates of the tongue contour of each frame of image are tracked, the change of the tongue contour is continuously recorded, and the tongue contour point set is fitted as a parameterized curve. By analyzing the parameterized curve of the tongue contour, the dynamic behavior data of the tongue contour is obtained. The neural network for regression task is used to extract the deep tongue tremor features related to the health status of the central nervous system, and is used together with the static tongue features and hyperspectral features for subsequent analysis of the fused tongue image feature data, realizing comprehensive consideration of the diagnosis information of the tongue health status, and helping to improve the fusion effect of the tongue features and the accuracy of the analysis result.

[0174] Specifically, in the embodiment, a deep neural network is used to extract features from the registered tongue region in the hyperspectral image, obtaining tongue spectral features, including:

[0175] The registered tongue region in the hyperspectral image is input into a three-dimensional convolutional neural network to extract deep features related to the chemical composition of the tongue, denoted as tongue spectral features.

[0176] It can be understood that, since the hyperspectral image has a three-dimensional data structure (two spatial dimensions and one spectral dimension), the three-dimensional convolutional neural network (3D-CNN) is used to extract features from the tongue region in the embodiment, which can process spatial and spectral information simultaneously, thereby extracting features related to chemical composition.

[0177] Optionally, the tongue spectral features extracted in the embodiment include but are not limited to the following:

[0178] 1. Spectral reflectance: The hyperspectral image records the reflectance of the object at different wavelengths, which is related to the chemical and physical properties of the object. By analyzing the spectral reflectance, the spectral features of the tongue tissue can be obtained.

[0179] 2. Spectral slope: The spectral slope can help identify the shape change of the spectral curve, reflecting the trend of the spectral feature change.

[0180] 3. Absorption features: Different chemical substances have specific absorption peaks at different wavelengths. By identifying these absorption peaks, the biochemical composition of the tongue tissue can be inferred.

[0181] S105, fuse the tongue spectral features, static tongue features and tongue tremor features to obtain fused tongue image feature data.

[0182] Optionally, the method of fusing the tongue spectral feature, the static tongue feature and the tongue tremor feature in the embodiment can adopt simple splicing, weighted summation or other more complex fusion strategies, which are not limited in the embodiment.

[0183] In S106, the deep neural network is used to analyze the fused tongue image feature data and predict the TCM analysis result of the current tongue image.

[0184] Specifically, the deep neural network is used to analyze the fused tongue image feature data and predict the TCM analysis result of the current tongue image, including:

[0185] The fused tongue image feature data is input into the neural network for classification task to obtain the probability of the TCM analysis result of the current tongue image belonging to different categories, and the category with the highest probability is selected as the prediction result of the current tongue image.

[0186] Optionally, the neural network for classification task in the embodiment can be a fully connected neural network, a convolutional neural network, a recurrent neural network, a deep belief network, etc., which are not limited in the embodiment.

[0187] For example, the implementation manner of inputting the fused tongue image feature data into the neural network for classification task to obtain the probability of the TCM analysis result of the current tongue image belonging to different categories is as follows: the fused tongue image feature data is input into the neural network for classification task, and it is assumed that the output is a vector z, where z e R K , K is the number of categories of the TCM analysis result. This vector z is the original prediction score of the neural network for each category of the TCM analysis result, and then, the softmax function is applied to the vector z so that the output can be interpreted as a probability distribution, the original prediction score is converted into the probability value of each category, and the sum of all probability values is 1.

[0188] Further, after obtaining the prediction result of the TCM analysis of the current tongue image, the result can be fed back to the remote server, the tongue surface multi-modal data can be displayed on the display in real time for the user to view, and the analysis result report of the final TCM tongue diagnosis index can be provided.

[0189] In another embodiment of the present specification, compared with the foregoing embodiment, the present embodiment increases the consideration of the visible light image of the tongue bottom when extracting the static tongue feature, wherein the implementation steps of amplifying the high-frequency component of the visible light image of the tongue bottom, extracting the tongue region, and image registration of the extracted tongue region are similar to the implementation steps of the visible light image in the foregoing embodiment, which will not be repeated here.

[0190] In this embodiment, the registered tongue region in the visible light image includes a tongue surface region image and a tongue base region image, and a deep neural network is used to extract features of the registered tongue region in the visible light image to obtain static tongue features, including:

[0191] The tongue surface region image and the tongue base region image are input into a double-branch convolutional neural network, and two branches extract features of the tongue surface region image and the tongue base region image respectively. One branch extracts tongue surface depth features related to color, texture, shape and local structure, and the other branch extracts tongue base depth features related to color, local structure and tissue structure.

[0192] The tongue surface depth features and the tongue base depth features are fused to obtain comprehensive tongue features, denoted as static tongue features.

[0193] The convolutional neural network branch architecture of the tongue surface region image is similar to the convolutional neural network architecture for extracting features of the registered tongue region in the visible light image in the foregoing embodiment, which will not be described here in this embodiment.

[0194] Since the features of the tongue base image can be more complex, including the tissue structure of the tongue base and possible lesion areas, the tongue base image needs deeper context information to identify more complex local structures. Preferably, in this embodiment, for the tongue base region image, the convolutional neural network architecture of EfficientNet or UNet can be considered. The specific reasons are as follows:

[0195] On the one hand, EfficientNet is a high-efficiency convolutional network that improves performance by scaling network width, depth and resolution, which is suitable for efficient feature extraction under limited computing resources. For the tongue base region image, EfficientNet can effectively capture global features in the tongue base region image while preserving details through multi-scale feature fusion, which is suitable for the relatively complex structure of the tongue base region.

[0196] On the other hand, UNet is a network architecture specifically designed for image segmentation, which has a symmetrical encoder-decoder structure. It can effectively extract local lesion features in the tongue base region image, while preserving multi-scale information through skip connections, especially suitable for tasks that require extraction of tongue base region features, such as lesion area detection. Moreover, UNet can preserve local details at different scales through its encoding-decoding structure, which is particularly suitable for areas with large local changes in the tongue base region image.

[0197] It can be understood that the visible light image of the tongue surface and the visible light image of the tongue bottom have significant differences in vision, for example, the tongue surface usually has more textures and local structure details such as tongue fur, cracks, and tongue veins, while the local structure of the tongue bottom image can be more complex, including the tissue structure of the tongue bottom, possible lesion areas, and the like, therefore, the embodiment adopts different convolutional neural network branch architectures for the tongue surface and tongue bottom images respectively to extract their local features, which can better capture the local structure and texture information of the tongue surface and tongue bottom images, and provide richer feature representations for subsequent classification or regression tasks, at the same time, the types of lesions that can appear in the tongue bottom region are different from those in the tongue surface region, therefore, the design of the double-branch convolutional neural network in the embodiment can better adapt to these differences and improve the prediction or identification ability of specific lesion types.

[0198] Referring to the accompanying drawings Figure 2 , Figure 2 A tongue image feature analysis system structure schematic diagram is provided for an embodiment of the present specification.

[0199] As Figure 2 shown, the tongue image feature analysis system can at least include an image acquisition module, an image processing module, a feature extraction module, a feature fusion module, and a feature analysis module, wherein:

[0200] The image acquisition module is configured to acquire tongue surface multi-modal image data, and the tongue surface multi-modal image data includes hyperspectral images, visible light images, and dynamic tongue images.

[0201] The image processing module is configured to perform high-frequency component amplification processing on the visible light images and the dynamic tongue images to obtain processed visible light images and dynamic tongue images, extract tongue region from the hyperspectral images, the processed visible light images, and the dynamic tongue images respectively, and perform image registration on the extracted tongue region.

[0202] The feature extraction module is configured to use a deep neural network to extract features of the registered tongue region in the hyperspectral images, the visible light images, and the dynamic tongue images respectively to obtain tongue spectral features, static tongue features, and tongue tremor features.

[0203] The feature fusion module is configured to fuse the tongue spectral features, the static tongue features, and the tongue tremor features to obtain fused tongue image feature data.

[0204] The feature analysis module is configured to use a deep neural network to analyze the fused tongue image feature data and predict the TCM analysis result of the current tongue image.

[0205] The image acquisition module is composed of three cameras, including a static image acquisition camera, a dynamic image acquisition camera and a multi-spectral camera, wherein the static image acquisition camera and the dynamic image acquisition camera are fixed at a preset position, and the hyperspectral camera is fixed on an electrically controlled sliding table for scanning imaging to obtain hyperspectral image data.

[0206] For example, when the face is placed in the collection position, the system broadcasts a voice prompt, controls the static image acquisition camera to collect a face image, controls the dynamic image acquisition camera to collect multiple tongue images at a fixed time interval t in sequence to form a dynamic tongue image sequence, and moves the sliding table to drive the hyperspectral camera to scan the tongue to obtain a hyperspectral image of the tongue.

[0207] It can be understood that the technical concept of the tongue image feature analysis system provided in the embodiment is similar to the technical concept of the tongue image feature analysis method provided in the foregoing embodiment, and the embodiment will not be described here.

[0208] The above is only the preferred embodiment of the present application and the explanation of the applied technical principles, and those skilled in the art should understand that the protection scope involved in the present disclosure is not limited to the technical solutions formed by the specific combination of the above technical features, and also covers other technical solutions formed by the combination of the above technical features or equivalent features without departing from the disclosed concept. For example, the above features are replaced with the technical features disclosed in the present disclosure (but not limited to) having similar functions to form a technical solution.

[0209] In addition, although each operation is described in a specific order, this should not be understood as requiring the operations to be performed in the specific order shown or in a sequential order. In certain circumstances, multitasking and parallel processing can be advantageous. Similarly, although specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Certain features described in the context of separate embodiments can also be combined in a single embodiment. Conversely, various features described in the context of a single embodiment can also be implemented separately or in any suitable subcombination.

Claims

1. A tongue image feature analysis method, characterized by, The method comprises the following steps: acquiring tongue surface multi-modal image data, the tongue surface multi-modal image data comprising hyperspectral images, visible light images, and dynamic tongue images; performing high-frequency component amplification processing on the visible light images and the dynamic tongue images to obtain processed visible light images and dynamic tongue images; extracting tongue regions from the hyperspectral images, the processed visible light images, and the dynamic tongue images respectively, and performing image registration on the extracted tongue regions; extracting features from the registered tongue regions in the hyperspectral images, the visible light images, and the dynamic tongue images respectively by using a deep neural network to obtain tongue spectral features, static tongue features, and tongue tremor features; fusing the tongue spectral features, the static tongue features, and the tongue tremor features to obtain fused tongue image feature data; analyzing the fused tongue image feature data by using a deep neural network and predicting a traditional Chinese medicine analysis result of a current tongue image; wherein the high-frequency component amplification processing on the visible light images to obtain the processed visible light images comprises: converting the visible light images to the frequency domain by using a discrete Fourier transform algorithm to obtain a frequency domain matrix of the visible light images; creating a high-pass mask and determining high-frequency components of the frequency domain matrix based on the high-pass mask; adding a phase shift to the high-frequency components to obtain a frequency domain matrix after adding a phase shift; calculating phase gradients of each frequency component in the horizontal direction and the vertical direction in the frequency domain matrix after adding a phase shift; determining phase changes of each frequency component based on the phase gradients of each frequency component; comparing the phase changes of each frequency component with a preset phase change threshold, and determining adjacent frequency components whose phase changes exceed the preset phase change threshold based on the comparison result; constructing a weight matrix based on a set of the adjacent frequency components whose phase changes exceed the preset phase change threshold; smoothing the weight matrix by using a two-dimensional Gaussian function; multiplying the smoothed weight matrix and the frequency domain matrix after adding a phase shift to obtain a final frequency domain matrix; performing inverse discrete Fourier transform on the final frequency domain matrix to obtain the processed visible light images.

2. The tongue image feature analysis method according to claim 1, wherein the extraction of the tongue regions from the hyperspectral images, the processed visible light images, and the dynamic tongue images respectively comprises: c. reading a plurality of frames of images from the processed dynamic tongue images in order to form a dynamic tongue image sequence. a. Selecting hyperspectral images The image of the Cth channel Based on a preset threshold, the image from the Cth channel... A mask is used to segment the corresponding tongue area. ; image of the Cth channel masking the tongue region point-wise multiplication, resulting in a hyperspectral image of the tongue region; b. the processed visible light image converted from the RGB space to the YCbCr space, an image of the Cr channel is selected , a mask corresponding to the tongue region is segmented from the image of the Cr channel according to a preset threshold . ; The image of the Cr channel is Masking the tongue region Point-wise multiplication, resulting in the tongue region in the visible light image; 3. The tongue image feature analysis method according to claim 2, wherein the feature extraction from the registered tongue region in the visible light images by using the deep neural network to obtain the static tongue features comprises: convert each image in the dynamic tongue image sequence from RGB space to YCbCr space, select the image in the Cr channel , segment a mask corresponding to the tongue region from the image in the Cr channel according to a preset threshold ; The image of the Cr channel is Masking the tongue region Point-wise multiplication, resulting in a tongue region in the dynamic tongue image. inputting the registered tongue region in the visible light images into a convolutional neural network to extract deep features related to color, texture, shape, and local structure, denoted as static tongue features.

4. The tongue image feature analysis method according to claim 3, wherein the registered tongue region in the visible light images comprises a tongue surface region image and a tongue bottom region image, and the feature extraction from the registered tongue region in the visible light images by using the deep neural network to obtain the static tongue features comprises: ​ ​ ​ The tongue surface area image and the tongue bottom area image are input into a double-branch convolutional neural network, and two branches respectively extract features of the tongue surface area image and the tongue bottom area image, one branch extracts tongue surface depth features related to color, texture, shape and local structure, and the other branch extracts tongue bottom depth features related to color, local structure and tissue structure; The tongue surface depth features and the tongue bottom depth features are fused to obtain comprehensive tongue features, which are denoted as static tongue features.

5. The tongue image feature analysis method of claim 2, wherein, a depth neural network is used to extract features of the registered tongue region in the dynamic tongue image, to obtain tongue tremor features, including: an edge detection is performed on the registered tongue region in the dynamic tongue image using a Sobel operator, to obtain a tongue contour of each image in the dynamic tongue image sequence; coordinate tracking is performed on the tongue contour of each image, to obtain a tongue contour point set of each image; the tongue contour point set of each image is fitted to obtain a tongue contour parameterized curve of each image; the tongue contour parameterized curves of all images are analyzed to obtain dynamic behavior data of the tongue contour; the dynamic behavior data of the tongue contour is input into a neural network for regression tasks, to extract depth features related to tremor amplitude, frequency, duration and tremor site, denoted as tongue tremor features.

6. The tongue image feature analysis method of claim 2, wherein, a depth neural network is used to extract features of the registered tongue region in the hyperspectral image, to obtain tongue spectral features, including: the registered tongue region in the hyperspectral image is input into a three-dimensional convolutional neural network, to extract depth features related to tongue chemical composition, denoted as tongue spectral features.

7. The tongue image feature analysis method of claim 1, wherein, a depth neural network is used to analyze the fused tongue image feature data and predict a traditional Chinese medicine analysis result of the current tongue image, including: the fused tongue image feature data is input into a neural network for classification tasks, to obtain probabilities of the current tongue image belonging to different categories of traditional Chinese medicine analysis results, and the category with the highest probability is selected as the prediction result of the current tongue image.

8. A tongue image feature analysis system characterized by, including: an image acquisition module for acquiring tongue surface multi-modal image data, the tongue surface multi-modal image data including a hyperspectral image, a visible light image and a dynamic tongue image; an image processing module for performing high-frequency component amplification processing on the visible light image and the dynamic tongue image, to obtain processed visible light images and dynamic tongue images, and extracting tongue regions from the hyperspectral image, the processed visible light image and the dynamic tongue image, and performing image registration on the extracted tongue regions; a feature extraction module for using a depth neural network to extract features of the registered tongue regions in the hyperspectral image, the visible light image and the dynamic tongue image, to obtain tongue spectral features, static tongue features and tongue tremor features; a feature fusion module for fusing the tongue spectral features, the static tongue features and the tongue tremor features to obtain fused tongue image feature data; The feature analysis module is configured to analyze the fused tongue image feature data by using a deep neural network, and predict a traditional Chinese medicine analysis result of the current tongue image. The image processing module is configured to perform the following steps: convert the visible light image to a frequency domain by using a discrete Fourier transform algorithm to obtain a frequency domain matrix of the visible light image; create a high-pass mask and determine high-frequency components of the frequency domain matrix based on the high-pass mask; add a phase shift to the high-frequency components to obtain a frequency domain matrix after adding the phase shift; calculate phase gradients of each frequency component in the frequency domain matrix after adding the phase shift along a horizontal direction and a vertical direction; determine a phase change of each frequency component based on the phase gradient of the frequency component; compare the phase change of each frequency component with a preset phase change threshold, and determine adjacent frequency components whose phase changes exceed the preset phase change threshold based on a comparison result; construct a weight matrix based on a set of the adjacent frequency components whose phase changes exceed the preset phase change threshold; smooth the weight matrix by using a two-dimensional Gaussian function; multiply the smoothed weight matrix with the frequency domain matrix after adding the phase shift to obtain a final frequency domain matrix; perform an inverse discrete Fourier transform on the final frequency domain matrix to obtain a processed visible light image.

Images