A method, device and medium for tongue image detection based on multi-source feature fusion

Abstract

This invention discloses a tongue image detection method, device, and medium based on multi-source feature fusion, comprising: Step 1, acquiring the subject's basic physiological indicators; Step 2, acquiring a facial image of the subject with their tongue protruding in a closed space; Step 3, preprocessing the facial image of the subject with their tongue protruding to segment the tongue image; Step 4, checking whether the quality of the tongue image meets the requirements; if yes, proceeding to Step 5; if no, returning to Step 2; Step 5, extracting the subject's tongue image features from the tongue image using a multi-branch multi-scale neural network; Step 6, extracting physiological indicator features from the basic physiological indicators, and combining the tongue image features and physiological indicator features to predict the subject's health status. This invention improves the accuracy of automated tongue image detection and reduces application costs.

Description

Technical Field

[0001] This invention relates to the field of tongue image detection technology, and in particular to a tongue image detection method, device and medium based on multi-source feature fusion. Background Technology

[0002] Traditional Chinese medicine (TCM), as one of the oldest treatment systems, is an important part of China's healthcare system. Tongue diagnosis is an important means in TCM for detecting a person's health status and discovering diseases, and it has the advantages of being low-cost, non-invasive, and radiation-free.

[0003] Both traditional Chinese medicine classics and modern medical research have demonstrated a strong correlation between the health status of specific human organs and tongue appearance characteristics. Taking liver disease as an example, some statistical studies in traditional Chinese medicine have revealed that frequently occurring tongue appearance characteristics in liver disease include red tongue, purple tongue, teeth marks, and yellow coating. These characteristics are important indicators of abnormal liver health. Modeling the mapping relationship between tongue images and specific diseases using machine learning methods to achieve automated image-based tongue detection will help enable rapid screening of specific diseases in large populations in areas with limited medical resources. However, existing automated tongue detection methods suffer from problems such as insufficient feature extraction and inappropriate feature usage, resulting in poor detection effectiveness and generalization performance.

[0004] Therefore, the industry needs a highly accurate automated tongue image detection technology that can be used for large-scale, low-cost disease screening. Summary of the Invention

[0005] In view of this, the purpose of this invention is to propose a tongue image detection method based on multi-source feature fusion, so as to improve the accuracy of automated tongue image detection and reduce application costs.

[0006] To achieve the above-mentioned technical objectives, the technical solution adopted by this invention is as follows:

[0007] This invention provides a tongue image detection method based on multi-source feature fusion, comprising the following steps:

[0008] Step 1: Obtain the basic physiological indicators of the subjects;

[0009] Step 2: Acquire facial images of the subject with their tongue sticking out in an enclosed space;

[0010] Step 3: Preprocess the image of the face with the tongue sticking out to segment out the tongue image;

[0011] Step 4: Check if the quality of the tongue image meets the requirements. If yes, proceed to step 5; otherwise, return to step 2.

[0012] Step 5: Use a multi-branch, multi-scale neural network to extract the subject's tongue features from the tongue image;

[0013] Step 6: Extract physiological indicator features from the basic physiological indicators, and combine the tongue appearance features and physiological indicator features to predict the health status of the subject.

[0014] Furthermore, step 1 specifically includes:

[0015] Step 11: Collect and record the basic physiological indicators of the subjects, including: gender, age, height, waist circumference, hip circumference, weight, systolic blood pressure and diastolic blood pressure;

[0016] Step 12: Represent the basic physiological indicators of the subjects in vector form as follows:

[0017] F = [f1, f2, f3, ..., f i ,…f k ],

[0018] Step 13: Divide the basic physiological indicators into categorical indicators and numerical indicators. The categorical indicators are generated using one-hot encoding, resulting in multiple f values ​​corresponding to the same number of categories as the categorical indicator. i Each of the numerical indicators corresponds to an f. i The total number of columns in the basic physiological index vector F is denoted as k.

[0019] Furthermore, step 2 specifically includes:

[0020] Step 21: A camera and light source are fixedly installed in an enclosed space;

[0021] Step 22: Adjust the camera's shooting angle and the position of the light source;

[0022] Step 23: Acquire facial images of the subject with their tongue sticking out using the camera and light source.

[0023] Furthermore, step 3 specifically includes:

[0024] Step 31: Use a target recognition network to identify the tongue-out facial image and extract the oral cavity region;

[0025] Step 32: Use an instance segmentation network to segment the tongue image from the oral cavity region;

[0026] Step 33: Fill the non-tongue regions in the segmented tongue image with black to obtain a standardized tongue image;

[0027] Step 34: Denote the pixel height of the standardized tongue image as H0 and the width as W0.

[0028] Furthermore, step 4 specifically includes:

[0029] Step 41: Calculate the maximum value of the height difference between the upper and lower edges of the non-black background pixels in the standardized tongue image:

[0030]

[0031] Among them, i j,1 This represents the index of the first row containing a non-zero pixel in the j-th column of a normalized tongue image, i. j,2 This represents the index of the second row containing a non-zero pixel in the j-th column of a normalized tongue image, i. j,1 and i j,2 ∈[0,H0-1], This indicates finding the maximum value;

[0032] Step 42: Determine the maximum value ΔH of the height position difference. max Is it less than the threshold H? s If yes, the standardized tongue image may have insufficient tongue protrusion, which is judged as too short, and the quality of the standardized tongue image does not meet the requirements; if no, proceed to step 43.

[0033] Step 43: Calculate the median of the left and right edge width position differences of the non-black background pixels in the standardized tongue image:

[0034]

[0035] Where, j i,1 j represents the index of the first column of the non-zero pixel in the i-th row of the normalized tongue image. i,2 j represents the index of the second column of the non-zero pixel in the i-th row of the standardized tongue image. i,1 and j i,2 ∈[0,W0-1], This indicates a search for the median.

[0036] Step 44: Determine the median ΔW of the width position difference. mid Is it less than the threshold W? s If yes, the standardized tongue image may have an unextensioned rolled tongue, which is judged as too narrow, and the quality of the standardized tongue image does not meet the requirements; if no, proceed to step 45.

[0037] Step 45: Calculate the average color value difference of the non-black background pixel regions in the standardized tongue image:

[0038]

[0039] Where, n C ΔC represents the total number of pixels with non-zero color values. i,jThe sum of color value differences between the pixel in row i and column j and its upper, lower, left, and right adjacent pixels is calculated using the following formula:

[0040]

[0041] Among them, C c,i,j This represents the RGB color value of the c-th channel of the pixel in the i-th row and j-th column;

[0042] Step 46: Determine the average value ΔC of the color value difference. mean Is it less than the threshold C? s If yes, the standardized tongue image may have motion blur, which is determined to be texture loss, and the quality of the standardized tongue image does not meet the requirements; if no, the quality of the standardized tongue image meets the requirements.

[0043] Furthermore, step 5 specifically includes:

[0044] Step 51: The multi-branch, multi-scale neural network extracts the tongue image feature vector f from the input standardized tongue image. img The multi-branch, multi-scale neural network comprises multiple branches, each branch containing one or more attention residual blocks connected in series, and the convolutional kernels of the attention residual blocks used in each branch contain multiple scales.

[0045] Step 52: Use ReLU and Sigmoid nonlinear activation functions to enhance the fitting ability of the input-output relationship of the multi-branch multi-scale neural network;

[0046] Step 53: Batch normalization normalizes the output at the current location;

[0047] Step 54: Apply the mask generated by the Mask R-CNN algorithm Feature maps applied to joint extraction of various branches In the diagram, C×H×W represents the number of channels, height, and width of the feature map output by the multi-branch, multi-scale neural network at this layer; the specific process is expressed by the formula:

[0048]

[0049] in, A mask representing the equivalent network image input size. A mask representing the equivalent feature size. This represents the feature map after eliminating excitation values ​​in non-tongue regions. BIL represents bimodal linear interpolation, MXP represents max pooling, and ⊙ represents multiplication via broadcasting.

[0050] Furthermore, step 6 specifically includes:

[0051] Step 61: Convert the tongue image feature vector f img The basic physiological indicator F is input into the multi-source feature fusion neural network;

[0052] Step 62: Divide the basic physiological index F into two groups. The first group of physiological index F1 is the tongue image feature vector f img Among the influential indicators, the second group of physiological indicators, F2, is an indicator that can indicate disease.

[0053] Step 63: Extract the feature vector f1 from the first set of physiological indicators F1, expressed by the formula:

[0054] f1 = MLP1(F1)

[0055] MLP1 represents a multilayer perceptron;

[0056] Step 64: Adjust the tongue image feature vector f after using feature vector f1. img The adjusted tongue image feature vector f' img Represented as:

[0057]

[0058] in, This indicates element-wise multiplication; Sigmoid represents the activation function.

[0059] Step 65: Extract the feature vector f2 from the second set of physiological indicators F2, expressed by the formula:

[0060] f2 = MLP2(F2)

[0061] MLP2 represents a multilayer perceptron;

[0062] Step 66: Combine the feature vector f2 with the adjusted tongue image feature vector f' img The merged data is used to output diagnostic results. This process can be expressed by the following formula:

[0063]

[0064] Here, FC represents a fully connected layer.

[0065] Furthermore, using whether a sample has a disease as the label y, both the multi-branch multi-scale neural network and the multi-source feature fusion neural network use the label y to calculate the weighted cross-entropy as the loss function L optimized during training.

[0066]

[0067]

[0068] Where r represents the number of categories of the predicted target, y i The label represents the ιth category. Let α represent the prediction for the i-th category. i k represents the weight of the i-th category. i is the number of samples of the i-th class in the training set, and β represents the hyperparameter used to adjust the degree of balance.

[0069] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement a tongue image detection method based on multi-source feature fusion as described above.

[0070] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the tongue image detection method based on multi-source feature fusion as described above.

[0071] By adopting the above technical solution, the present invention has the following beneficial effects compared with the prior art:

[0072] 1. This invention preprocesses the collected facial images of tongue protrusion to obtain standardized tongue images, eliminating the interference of poor image quality and irrelevant information on tongue feature extraction, and has high robustness and generalization ability.

[0073] 2. This invention automatically focuses on tongue features associated with the prediction target by training a neural network, eliminating the need for tedious manual annotation of key areas, thus improving application efficiency and reducing application costs.

[0074] 3. The multi-branch, multi-scale neural network used in this invention can extract multi-scale feature information from tongue images, which helps to fully extract tongue image features.

[0075] 4. This invention combines tongue appearance features and physiological indicators to output comprehensive diagnostic results, which can improve the predictive accuracy of automated tongue appearance detection.

[0076] 5. The automated tongue image detection of the present invention has high efficiency, and the prediction time for each subject is within 50 milliseconds, which can meet the needs of large-scale population screening. Attached Figure Description

[0077] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0078] Figure 1 This is an execution flowchart of a tongue image detection method based on multi-source feature fusion provided in an embodiment of the present invention.

[0079] Figure 2 This is a system structure diagram provided in an embodiment of the present invention.

[0080] Figure 3 This is a schematic diagram of the preprocessing process of a standardized tongue image provided in an embodiment of the present invention.

[0081] Figure 4 This is a flowchart of tongue image quality inspection provided in an embodiment of the present invention.

[0082] Figure 5 This is a schematic diagram of the structure of a multi-branch, multi-scale neural network provided in an embodiment of the present invention.

[0083] Figure 6 This is a schematic diagram of the structure of the multi-source feature fusion neural network provided in the embodiment of the present invention.

[0084] Figure 7 This is a schematic diagram of an electronic device provided in an embodiment of the present invention.

[0085] Figure 8 This is a schematic diagram of a computer-readable storage medium provided in an embodiment of the present invention. Detailed Implementation

[0086] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be particularly noted that the following embodiments are for illustrative purposes only and do not limit the scope of the invention. Similarly, the following embodiments are only some, not all, embodiments of the present invention, and all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0087] Please see Figures 1-6 The present invention provides a tongue image detection method based on multi-source feature fusion, comprising the following steps:

[0088] Step 1: Obtain the basic physiological indicators of the subjects;

[0089] In this embodiment, step 1 specifically includes:

[0090] Step 11: Collect and record the basic physiological indicators of the subjects, including: gender, age, height, waist circumference, hip circumference, weight, systolic blood pressure and diastolic blood pressure; the key feature is that they can be easily obtained at low cost through non-invasive methods.

[0091] Step 12: Represent the basic physiological indicators of the subjects in vector form as follows:

[0092] F = [f1, f2, f3, ..., f i ,…f k ],

[0093] Step 13: Divide the basic physiological indicators into categorical indicators and numerical indicators. The categorical indicators are generated using one-hot encoding (also known as one-hot encoding, a method of converting categorical variables into binary vectors) to produce multiple f values ​​with the same number of categories as the corresponding categorical indicator. i Each of the numerical indicators corresponds to an f. i The total number of columns in the basic physiological index vector F is denoted as k.

[0094] For example, gender is a categorical indicator, which can be divided into male or female, so the number of categories is 2, resulting in f1 and f2. Age, height, waist circumference, hip circumference, weight, systolic blood pressure, and diastolic blood pressure are all numerical indicators. For a subject, age, height, waist circumference, hip circumference, weight, systolic blood pressure, and diastolic blood pressure have only one corresponding value. Therefore, age corresponds to f3, height corresponds to f4, waist circumference corresponds to f5, hip circumference corresponds to f6, weight corresponds to f7, systolic blood pressure corresponds to f8, and diastolic blood pressure corresponds to f9. In this case, the total number of columns k of the basic physiological indicator vector F is 9.

[0095] Step 2: Acquire facial images of the subject with their tongue sticking out in an enclosed space;

[0096] In this embodiment, step 2 specifically includes:

[0097] Step 21: A camera and light source are fixedly installed in an enclosed space;

[0098] Step 22: Adjust the shooting angle of the camera and the position of the light source. The clarity and stability of the captured image can be increased by adjusting the camera and the light source.

[0099] Step 23: Acquire facial images of the subject with their tongue sticking out using the camera and light source.

[0100] Step 3: Preprocess the image of the face with the tongue sticking out to segment out the tongue image;

[0101] In this embodiment, step 3 specifically includes:

[0102] Step 31: Use a target recognition network to identify the tongue-out facial image and extract the oral cavity region;

[0103] Step 32: Use an instance segmentation network to segment the tongue image from the oral cavity region;

[0104] Step 33: Fill the non-tongue regions in the segmented tongue image with black to obtain a standardized tongue image. This is because the tongue occupies only a small area in the original captured image of the tongue protruding face. If the instance segmentation network is directly used to generate a mask for the tongue region, the computational cost for network training and prediction is very high, and the accuracy is also poor. The target recognition network can be the lightweight YoloX, and the instance segmentation network can be the more accurate Mask R-CNN. The preprocessing process for the standardized tongue image is as follows: Figure 3 As shown;

[0105] Step 34: Denote the pixel height of the standardized tongue image as H0 and the width as W0.

[0106] Step 4: Check if the quality of the tongue image meets the requirements. If yes, proceed to step 5; otherwise, return to step 2.

[0107] In this embodiment, the process of checking the quality of the tongue image in step 4 is as follows: Figure 4 As shown, it specifically includes:

[0108] Step 41: Calculate the maximum value of the height difference between the upper and lower edges of the non-black background pixels in the standardized tongue image:

[0109]

[0110] Among them, i j,1 This represents the index of the first row containing a non-zero pixel in the j-th column of a normalized tongue image, i. j,2 This represents the index of the second row containing a non-zero pixel in the j-th column of a normalized tongue image, i. j,1 and i j,2 ∈[0,H0-1], This indicates finding the maximum value;

[0111] Step 42: Determine the maximum value ΔH of the height position difference. max Is it less than the threshold H? s If yes, the standardized tongue image may have insufficient tongue protrusion, which is judged as too short, and the quality of the standardized tongue image does not meet the requirements; if no, proceed to step 43.

[0112] Step 43: Calculate the median of the left and right edge width position differences of the non-black background pixels in the standardized tongue image:

[0113]

[0114] Where, j i,1 j represents the index of the first column of the non-zero pixel in the i-th row of the normalized tongue image. i,2j represents the index of the second column of the non-zero pixel in the i-th row of the standardized tongue image. i,1 and j i,2 ∈[0,W0-1], This indicates a search for the median.

[0115] Step 44: Determine the median ΔW of the width position difference. mid Is it less than the threshold W? s If yes, the standardized tongue image may have an unextensioned rolled tongue, which is judged as too narrow, and the quality of the standardized tongue image does not meet the requirements; if no, proceed to step 45.

[0116] Step 45: Calculate the average color value difference of the non-black background pixel regions in the standardized tongue image:

[0117]

[0118] Where, n C ΔC represents the total number of pixels with non-zero color values. i,j The sum of color value differences between the pixel in row i and column j and its upper, lower, left, and right adjacent pixels is calculated using the following formula:

[0119]

[0120] Among them, C c,i,j This represents the RGB color value of the c-th channel of the pixel in the i-th row and j-th column;

[0121] Step 46: Determine the average value ΔC of the color value difference. mean Is it less than the threshold C? s If yes, the standardized tongue image may have motion blur, which is determined to be texture loss, and the quality of the standardized tongue image does not meet the requirements; if no, the quality of the standardized tongue image meets the requirements.

[0122] Therefore, tongue images deemed too short, too narrow, or lacking texture are classified as poor quality. Three thresholds H are used. s W s and C s It can be adjusted according to the camera's focal length, shooting distance, and lighting conditions.

[0123] Step 5: Use a multi-branch, multi-scale neural network to extract the subject's tongue features from the tongue image;

[0124] In this embodiment, step 5 specifically includes:

[0125] Step 51: The multi-branch, multi-scale neural network extracts the tongue image feature vector f from the input standardized tongue image. imgThe specific structure of the multi-branch, multi-scale neural network is as follows: Figure 5 As shown, the multi-branch multi-scale neural network comprises multiple branches, each containing one or more attention residual blocks connected in series. These attention residual blocks are classified into basic attention residual blocks and bottleneck attention residual blocks according to different convolutional layer settings, and are labeled with suffixes -3, -5, and -7 based on the kernel scale of the non-1×1 convolutional layers. The convolutional kernels of the attention residual blocks used in each branch contain multiple scales, thus having different fields of view and focusing on extracting tongue image feature information at the corresponding scale.

[0126] Step 52: Use ReLU and Sigmoid nonlinear activation functions to enhance the fitting ability of the input-output relationship of the multi-branch multi-scale neural network; in the figure, ReLU and Sigmoid are nonlinear activation functions used to enhance the network's ability to fit complex input-output relationships.

[0127] Step 53: Batch normalization normalizes the output at the current location to prevent excessively large values ​​and gradient vanishing.

[0128] Step 54: Apply the mask generated by the Mask R-CNN algorithm (Mask R-CNN is an instance segmentation algorithm that performs segmentation based on object detection, and it is a method that outputs a high-quality instance segmentation mask while effectively detecting objects). Feature maps applied to joint extraction of various branches In order to eliminate the interference of excitation values ​​in non-tongue regions, C×H×W represents the number of channels, height, and width of the output feature map of the multi-branch multi-scale neural network at this layer; the specific process is expressed by the formula:

[0129]

[0130] in, A mask representing the equivalent network image input size. A mask representing the equivalent feature size. This represents the feature map after eliminating excitation values ​​in non-tongue regions. BIL represents bimodal linear interpolation, MXP represents max pooling, and ⊙ represents multiplication via broadcasting.

[0131] Step 6: Extract physiological indicator features from the basic physiological indicators, and combine the tongue appearance features and physiological indicator features to predict the health status of the subject.

[0132] In this embodiment, step 6 specifically includes:

[0133] Step 61: Convert the tongue image feature vector f imgThe basic physiological indicator F is input into the multi-source feature fusion neural network; the specific structure of the multi-source feature fusion neural network is as follows: Figure 6 As shown;

[0134] Step 62: Divide the basic physiological index F into two groups. The first group of physiological index F1 is the tongue image feature vector f img Influential indicators include gender and age; the second group of physiological indicators, F2, are indicators that can indicate disease, including height, waist circumference, hip circumference, weight, systolic blood pressure, and diastolic blood pressure.

[0135] Step 63: Extract the feature vector f1 from the first set of physiological indicators F1, expressed by the formula:

[0136] f1 = MLP1(F1)

[0137] MLP1 represents a multilayer perceptron;

[0138] Step 64: Adjust the tongue image feature vector f after using feature vector f1. img The adjusted tongue image feature vector f' img Represented as:

[0139]

[0140] in, This indicates element-wise multiplication; Sigmoid represents the activation function.

[0141] Step 65: Extract the feature vector f2 from the second set of physiological indicators F2, expressed by the formula:

[0142] f2 = MLP2(F2)

[0143] MLP2 represents a multilayer perceptron;

[0144] Step 66: Combine the feature vector f2 with the adjusted tongue image feature vector f' img The merged data is used to output diagnostic results. This process can be expressed by the following formula:

[0145]

[0146] Here, FC represents a fully connected layer.

[0147] In this embodiment, whether a sample has a disease is used as the label y. Both the multi-branch multi-scale neural network and the multi-source feature fusion neural network use the label y to calculate the weighted cross-entropy as the loss function L optimized during training.

[0148]

[0149] Where r represents the number of categories of the predicted target, y i This represents the label of the i-th category. Let α represent the prediction for the i-th category. i k represents the weight of the i-th category. i is the number of samples of the i-th class in the training set, and β represents the hyperparameter used to adjust the degree of balance.

[0150] like Figure 7 As shown, this embodiment of the invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the above-described tongue image detection method based on multi-source feature fusion.

[0151] like Figure 8 As shown, this embodiment of the invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described tongue image detection method based on multi-source feature fusion.

[0152] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0153] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0154] The above description is only a part of the embodiments of the present invention and does not limit the scope of protection of the present invention. Any equivalent device or equivalent process transformation made based on the content of the present invention specification and drawings, or direct or indirect application in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A tongue image detection method based on multi-source feature fusion, characterized in that, Includes the following steps: Step 1: Obtain the basic physiological indicators of the subjects; Step 2: Acquire facial images of the subject with their tongue sticking out in an enclosed space; Step 3: Preprocess the tongue-out facial image to segment the tongue image; Step 4: Check if the quality of the tongue image meets the requirements. If yes, proceed to step 5; otherwise, return to step 2. Step 5: Extract tongue features from the subject's tongue image using a multi-branch, multi-scale neural network; specifically including: Step 51: The multi-branch, multi-scale neural network extracts tongue image feature vectors from the input standardized tongue image. The multi-branch, multi-scale neural network comprises multiple branches, each branch containing one or more attention residual blocks connected in series, and the convolutional kernels of the attention residual blocks used in each branch contain multiple scales. Step 52: Use ReLU and Sigmoid nonlinear activation functions to enhance the fitting ability of the input-output relationship of the multi-branch multi-scale neural network; Step 53: Batch normalization normalizes the output at the current location; Step 54: Apply the mask generated by the Mask R-CNN algorithm Feature maps applied to joint extraction of various branches middle, This represents the number of channels, height, and width of the feature map output by a multi-branch, multi-scale neural network; the specific process is expressed by the following formula: ， ， ， in, A mask representing the equivalent network image input size. A mask representing the equivalent feature size. This represents the feature map after eliminating excitation values ​​from non-tongue regions. Indicates bimodal linear interpolation. This indicates max pooling. This indicates that multiplication is performed via a broadcast mechanism; Step 6: Extract physiological indicator features from the basic physiological indicators, and combine the tongue appearance features and physiological indicator features to predict the subject's health status; specifically including: Step 61: Convert the tongue image feature vector and basic physiological indicators The input is fed into a multi-source feature fusion neural network; Step 62: Transfer the basic physiological indicators Divided into two groups, the first group's physiological indicators tongue image feature vector Influential indicators, second group of physiological indicators These are indicators that can suggest the nature of a disease; Step 63: From the first set of physiological indicators Extracting feature vectors This can be expressed as a formula: in, Represents a multilayer perceptron; Step 64: Through the feature vector Adjusting tongue image feature vector Adjusted tongue image feature vector Represented as: in, This indicates element-wise multiplication. Indicates the activation function; Step 65, from the second group of physiological indicators Extracting feature vectors This can be expressed as a formula: in, Represents a multilayer perceptron; Step 66: Transfer the feature vector With the adjusted tongue image feature vector The merged data is used to output diagnostic results. This process can be expressed by the following formula: in, This indicates a fully connected layer.

2. The tongue image detection method based on multi-source feature fusion as described in claim 1, characterized in that, Step 1 specifically includes: Step 11: Collect and record the basic physiological indicators of the subjects, including: gender, age, height, waist circumference, hip circumference, weight, systolic blood pressure and diastolic blood pressure; Step 12: Represent the basic physiological indicators of the subjects in vector form as follows: , Step 13: Divide the basic physiological indicators into categorical indicators and numerical indicators. The categorical indicators are generated using one-hot encoding, resulting in multiple categories with the same number of corresponding categories. Each of the numerical indicators corresponds to one Basic physiological index vector The total number of columns is denoted as .

3. The tongue image detection method based on multi-source feature fusion as described in claim 1, characterized in that, Step 2 specifically includes: Step 21: A camera and light source are fixedly installed in an enclosed space; Step 22: Adjust the camera's shooting angle and the position of the light source; Step 23: Acquire facial images of the subject with their tongue sticking out using the camera and light source.

4. The tongue image detection method based on multi-source feature fusion as described in claim 1, characterized in that, Step 3 specifically includes: Step 31: Use a target recognition network to identify the tongue-out facial image and extract the oral cavity region; Step 32: Use an instance segmentation network to segment the tongue image from the oral cavity region; Step 33: Fill the non-tongue regions in the segmented tongue image with black to obtain a standardized tongue image; Step 34: Record the pixel height of the standardized tongue image as... Width is denoted as .

5. The tongue image detection method based on multi-source feature fusion as described in claim 4, characterized in that, Step 4 specifically includes: Step 41: Calculate the maximum value of the height difference between the upper and lower edges of the non-black background pixels in the standardized tongue image: in, Represents the first normalized tongue image The index of the first row containing non-zero pixels in the column. Represents the first normalized tongue image The index of the second row containing non-zero pixels in the column. and , This indicates finding the maximum value; Step 42: Determine the maximum value of the height position difference. Is it less than the threshold? If yes, the standardized tongue image may have insufficient tongue protrusion, which is judged as too short, and the quality of the standardized tongue image does not meet the requirements; if no, proceed to step 43. Step 43: Calculate the median of the left and right edge width position differences of the non-black background pixels in the standardized tongue image: in, Represents the first normalized tongue image The index of the first column containing the non-zero pixel in the row. Represents the first normalized tongue image The index of the second column containing the non-zero pixels in the row. and , This indicates a search for the median. Step 44: Determine the median of the width position difference. Is it less than the threshold? If yes, the standardized tongue image may have an unextensioned rolled tongue, which is judged as too narrow, and the quality of the standardized tongue image does not meet the requirements; if no, proceed to step 45. Step 45: Calculate the average color value difference of the non-black background pixel regions in the standardized tongue image: in, This represents the total number of pixels with non-zero color values. Indicates the first Line 1 The sum of the color value differences between a column pixel and its upper, lower, left, and right adjacent pixels is calculated using the following formula: in, Indicates the first Line 1 The first column of pixels channel Color value; Step 46: Determine the average value of the color value difference. Is it less than the threshold? If yes, the standardized tongue image may have motion blur, which is determined to be texture loss, and the quality of the standardized tongue image does not meet the requirements; if no, the quality of the standardized tongue image meets the requirements.

6. The tongue image detection method based on multi-source feature fusion as described in claim 1, characterized in that, Using whether the sample has a disease as a label Both the multi-branch multi-scale neural network and the multi-source feature fusion neural network use labels. Calculate the weighted cross-entropy as the loss function to be optimized during training. : in, This represents the number of categories of the predicted target. Indicates the first Category labels, Indicates the first Category prediction Indicates the first Category weights It is the first in the training set Number of samples in each category This represents the hyperparameter used to adjust the degree of balance.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements a tongue image detection method based on multi-source feature fusion as described in any one of claims 1 to 6.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by the processor, the program implements a tongue image detection method based on multi-source feature fusion as described in any one of claims 1 to 6.

Images