Physique and visceral organ identification method based on multi-domain fusion

Abstract

The invention discloses a constitution and visceral organ identification method based on multi-domain fusion, and the method mainly comprises the steps: extracting chaotic domain features and frequency domain features through a multi-domain fusion MLP model after obtaining a tongue image, and carrying out the fusion; and then, according to the fusion features, generating a constitution type prediction vector and an organ type prediction vector. The deep fusion feature information of the chaos domain and the discrete Fourier transform domain of the tongue image is extracted innovatively by adopting the multi-domain fusion type MLP network, so that the pathological information of the tongue image is represented comprehensively, and the multi-task recognition performance of the physique and the visceral organs of the model is improved.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically to a method for identifying constitution and organs based on multi-domain fusion. Background Technology

[0002] With the expanding application of artificial intelligence technology in the field of TCM-assisted diagnosis, constitution and organ joint recognition (BCR) based on computer vision has become a research hotspot. This task is essentially a multi-label recognition problem, usually using visual images such as the tongue and face as input, aiming to assist TCM doctors in initial diagnosis by intelligently identifying constitution type and organ status.

[0003] Currently, although MLP-like architectures have become an important network paradigm besides CNN and Transformer, no research has yet explored their application potential in BCR tasks.

[0004] Although Global Filter Network (GFNet) proposes a learnable global filter for frequency-spatial feature extraction, its frequency domain transformation is limited to the basic Fourier transform level. When the input samples have blurred boundaries (such as two types of tongue images with similar textures but different labels), GFNet has difficulty capturing subtle differences.

[0005] The sensitivity of chaotic transformations to initial conditions can effectively amplify subtle differences between similar samples, but existing methods completely ignore their value in feature enhancement. Traditional models cannot utilize chaotic dynamics to distinguish visually similar but pathologically different tongue appearances, leading to a bottleneck in recognition accuracy. Summary of the Invention

[0006] In view of this, in order to at least partially solve the above-mentioned technical problems, the present invention provides a constitution and organ identification method based on multi-domain fusion, which aims to fully explore the complementarity of chaotic transformation and discrete Fourier transform (DFT): DFT describes global texture, chaotic transformation captures local nonlinear dynamics, and the combination of the two can comprehensively characterize the pathological information of tongue appearance.

[0007] In order to achieve the above object, the present invention adopts the following technical solutions:

[0008] A method for identifying constitution and organs based on multi-domain fusion, comprising the following steps:

[0009] Acquire tongue images, extract chaotic domain features and frequency domain features using a multi-domain fusion MLP model, and then fuse them.

[0010] Generate body type prediction vectors and organ category prediction vectors based on the fusion features.

[0011] In one optional embodiment, the multi-domain fusion MLP model includes a preliminary feature extraction unit, multiple end-to-end connected multi-domain fusion downsampling units, and a fully connected layer.

[0012] In one optional embodiment, the multi-domain fusion downsampling unit includes an MDF block, a downsampling layer, and a convolutional layer;

[0013] MDF blocks are used to extract and fuse chaotic domain features and frequency domain features from the feature sequence in the preliminary features from the two dimensions of height and width.

[0014] In one optional embodiment, the chaotic domain feature extraction step is as follows:

[0015] The input features are subjected to chaotic transformation sequentially through linear transformation, normalization layer and activation function;

[0016] Φ = ReLU(BN(Lin(x))

[0017] Where x represents the input feature;

[0018] The transformed features are mapped using Logistic regression to obtain chaotic domain features; the mapping method is as follows:

[0019] Φ chaos =μ·Φ(1-Φ)

[0020] In the formula, Φ represents the chaotic transformation characteristic, Φ chaos denoted by , which represents the characteristics of the chaotic domain, and μ represents the chaotic parameter.

[0021] In one optional embodiment, the frequency domain feature extraction formula is:

[0022]

[0023] In the formula, x represents the input feature, DFT[·] represents the two-dimensional discrete Fourier transform function, iDFT[·] represents the two-dimensional inverse discrete Fourier transform function, and H d This represents the learnable weights.

[0024] In one optional embodiment, the chaotic domain features and frequency domain features are concatenated, and feature fusion is achieved through a linear layer, as expressed in the formula:

[0025]

[0026] Where Cat[·] represents the feature concatenation operation, and Lin(·) represents a linear layer, preferably a convolutional layer with a 1×1 kernel. Represents frequency domain characteristics, Φ chaos This represents the characteristics of a chaotic domain.

[0027] In one alternative embodiment, a residual connection is made with the input feature x during fusion.

[0028] In one optional embodiment, the tongue image is acquired and then segmented into blocks; preliminary features of each block are extracted to form a corresponding feature sequence; and multiple feature sequences are input into a multi-domain fusion MLP model.

[0029] The feature fusion also includes:

[0030] Aggregate the fused features of each sub-feature in the feature sequence;

[0031] The aggregated features of each feature sequence are aggregated again to obtain the fused features of the tongue image.

[0032] Based on key location feature loss, the multi-domain fusion MLP model is trained and optimized:

[0033] L total =λ·L BCE +(1-λ)·(L con +L loc )

[0034] Where, L total L represents the loss function for training. BCE This represents the binary cross-entropy loss, where λ is the smoothing coefficient, and L... con Key location loss of tongue image representing body type, L loc Key location loss of tongue images representing organ categories.

[0035] The present invention provides a method for identifying constitution and organs based on multi-domain fusion, which, compared with existing technologies, offers the following advantages:

[0036] 1) Innovatively employing a multi-domain fusion-type MLP network to extract deep fusion feature information from the chaotic domain and discrete Fourier transform domain of tongue images, thereby improving the model's multi-task recognition performance of physical constitution and organs;

[0037] 2) The multi-label smoothing position label loss function is used as the objective function of the MDF-MLP model. The key position loss of the tongue image corresponding to the body type and the key position loss of the tongue image corresponding to the organ type are calculated and incorporated into the overall loss through the smoothing coefficient, thereby guiding the training of the MDF-MLP model. Attached Figure Description

[0038] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0039] Figure 1This is a schematic diagram of the multi-domain fusion MLP model structure of the present invention;

[0040] Figure 2 This is a schematic diagram of the MDF block structure of the present invention;

[0041] Figure 3 This is a schematic diagram of the Chaos-DFT module structure of the present invention. Detailed Implementation

[0042] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0043] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0044] To overcome the bottlenecks of existing technologies, this invention provides a method for identifying constitution and organs based on multi-domain fusion, considering the chaotic domain features and frequency domain features of tongue images. In this embodiment, the identification steps include:

[0045] Acquire tongue images, extract chaotic domain features and frequency domain features using a multi-domain fusion MLP model, and then fuse them.

[0046] Generate body type prediction vectors and organ category prediction vectors based on the fusion features.

[0047] Among them, the constitution types include balanced constitution, qi deficiency constitution, yang deficiency constitution, yin deficiency constitution, phlegm-dampness constitution, damp-heat constitution, blood stasis constitution, qi stagnation constitution, and special constitution;

[0048] The organ categories include large intestine, gallbladder, lungs, liver, bladder, spleen, kidneys, stomach, small intestine, and heart.

[0049] To further understand the features, technical means, and specific objectives and functions achieved by the present invention, the present invention will be described in further detail below with reference to the accompanying drawings and specific embodiments.

[0050] Example 1:

[0051] In this embodiment, a multi-domain fusion MLP model is first constructed. This model includes sequentially connected preliminary feature extraction units, multiple end-to-end connected multi-domain fusion downsampling units, and a fully connected layer. (Structure reference...) Figure 1 ;

[0052] After obtaining the tongue image, this application first scales it to a standard size of 256×256, then randomly crops a smaller image of 224×224 from the scaled image, and then randomly flips the cropped image horizontally to perform data augmentation.

[0053] Then, patch embedding is used to divide the input tongue image into blocks, and a preliminary feature extraction unit is used to extract preliminary features from each image block. Preferably, the preliminary feature extraction unit is a convolutional layer with a 7×7 kernel, a stride of 4, and padding of 2. After extracting the preliminary features of each image block and obtaining the corresponding feature sequence, multiple feature sequences are input into a multi-domain fusion MLP model.

[0054] Furthermore, the multi-domain fusion downsampling unit includes an MDF block, a downsampling layer, and a convolutional layer; the MDF block is used to extract and fuse the chaotic domain features and frequency domain features of the feature sequence from the initial features in the two dimensions of height and width; the downsampling layer is used to reduce the interaction transformation of width and height features.

[0055] This embodiment sets up four stages. In the last stage, the downsampling layer and convolutional layer are directly replaced with a fully connected (FC) layer to directly output the predicted vectors for body constitution and organ categories. The feature dimensions of each stage are as follows: and And C1 = 64, C2 = 128, C3 = 320, and C4 = 512.

[0056] In one embodiment, the structure of the MDF block is as follows: Figure 2 As shown, it includes an H (height) feature extraction branch, a W (width) feature extraction branch, and a C (channel) branch; it should be noted that this embodiment only extracts depth features in the two dimensions of height and width, and the feature extraction process of the two branches is the same.

[0057] Taking the H (height) feature extraction branch as an example, the extraction process is as follows: Figure 3 As shown:

[0058] Regarding the characteristics of the chaotic domain:

[0059] The input features are subjected to chaotic transformation sequentially through linear transformation, normalization layer, and activation function; the formula is expressed as:

[0060] Φ H =ReLU(BN(Lin(x) H ))

[0061] Where, x HRepresents the vector of features in the feature sequence in terms of height; ReLU represents a non-linear activation function; BN represents a batch normalization layer; Lin represents a linear layer, implemented using a convolutional layer with a 1×1 kernel, used to improve the non-linearity of channel characteristics.

[0062] Then, the transformed features are mapped based on Logistic regression to obtain chaotic domain features; the mapping method is as follows:

[0063] Φ H,chaos =μ·Φ H (1-Φ H

[0064] In the formula, Φ H Φ represents the chaotic transformation characteristics on the height channel. H,chaos This represents the chaotic domain characteristics on the height channel. μ represents the chaos parameter, which is usually set to 4 according to the Lyapunov exponent theory.

[0065] For frequency domain features, the formula is expressed as:

[0066]

[0067] In the formula, Represents frequency domain characteristics, DFT[·] denotes the two-dimensional discrete Fourier transform function, iDFT[·] denotes the two-dimensional inverse discrete Fourier transform function, H dh This represents the learnable weights at different heights.

[0068] Furthermore, the chaotic domain features and frequency domain features are concatenated, and feature fusion is achieved through a linear layer:

[0069]

[0070] Where Cat[·] represents the feature concatenation operation, and Lin(·) represents a linear layer, preferably a convolutional layer with a 1×1 kernel. Φ represents the frequency domain characteristics of the height channel. H,chaos This represents the chaotic domain features in the height channel. Preferably, during fusion, a residual connection is made with the input feature x, i.e.

[0071] Y = MDF FC (BN(x))+x

[0072] In one embodiment, feature fusion further includes:

[0073] Aggregate the fused features of each sub-feature in the feature sequence; assuming the input feature sequence is X = [x1, x2, ..., x...] n The aggregated features are as follows:

[0074]

[0075] Represents the set of multi-domain fusion sub-features. i represents the index number of the sub-feature in the feature sequence; W to Represents the weight parameters. This represents the weight parameters corresponding to the i-th and j-th feature sequences.

[0076] The aggregated features of each feature sequence are aggregated again to obtain the fused features of the tongue image;

[0077]

[0078] In the formula, j represents the index of the feature sequence (image patch), and m represents the number of feature sequences. This represents the weight of the j-th feature sequence.

[0079] Finally, based on the fusion features, predictive vectors for body type and organ category are generated.

[0080] In this embodiment, a binary classifier is used to input the feature C(X) into the fully connected layer FC, which is implemented by the sigmoid activation function σ(·). The body type is as follows:

[0081]

[0082] In the formula, This indicates the parameters of the FC layer output in the constitution type task. Represents the entire parameter set; The output vector for constitution recognition consists of the probability of each constitution type category label. composition.

[0083] Similarly, a binary classifier is used to input the feature C(X) into the FC layer, which is implemented by the sigmoid activation function σ(·). The organ categories are as follows:

[0084]

[0085] In the formula, This represents the parameters of the output FC layer in the organ category recognition task; Q represents the entire parameter set; loc (X,θ loc ) represents the output vector for internal organ identification, consisting of the probability p(q) of each internal organ. i |X,θ loc )composition.

[0086] In a preferred embodiment, when outputting the corresponding constitution label and organ category label, a residual connection is performed with the fusion features of the tongue image, that is:

[0087] Z = CH FC (BN(C(x)))+C(X)

[0088] Z represents the constitution label and organ category label corresponding to the input visual diagnosis image.

[0089] Example 2:

[0090] The multi-domain fusion MLP model of this invention is pre-trained in the following manner:

[0091] A dataset was constructed by collecting tongue images, labeling them with body constitution and organ tags, and then constructing a dataset for training and testing multi-domain fusion MLP models.

[0092] The overall loss is calculated using key location features of body type and organ category from tongue images, and then used to train a multi-domain fusion MLP network model. The expression for the multi-label smoothing location label loss function is as follows:

[0093] L total =λ·L BCE +(1-λ)·(L con +L loc )

[0094] Where, L total L represents the loss function for training. BCE This represents the binary cross-entropy loss, where λ is the smoothing coefficient, and L... con Key location loss of tongue image representing body type, L loc Key location loss of tongue images representing organ categories.

[0095] In one embodiment, the training method is as follows: using the deep learning framework PyTorch and the model library timm, the MDF-MLP network model is trained using the stochastic gradient descent (SGD) optimizer. The learning rate of SGD is dynamically decayed from the initial learning rate to the minimum learning rate using a cosine annealing strategy. The input image size is scaled to 224×224 using the center cropping data augmentation method, and the training and test images are normalized. The specific parameters are shown in the table below.

[0096] parameter Value SGD Momentum 0.9 Learning rate weight decay 5e-4 Batch size 32 Initial learning rate 1e-3 Minimum learning rate 2.5e-4 Input image size 224×224 Model training epochs 200

[0097] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0098] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for identifying constitution and organs based on multi-domain fusion, characterized in that, Acquire tongue images, extract chaotic domain features and frequency domain features using a multi-domain fusion MLP model, and then fuse them. Generate body type prediction vectors and organ category prediction vectors based on the fusion features.

2. The method for identifying body constitution and organs according to claim 1, characterized in that, The multi-domain fusion MLP model includes a preliminary feature extraction unit, multiple multi-domain fusion downsampling units connected end-to-end, and a fully connected layer.

3. The method for identifying body constitution and organs according to claim 2, characterized in that, The multi-domain fusion downsampling unit includes an MDF block, a downsampling layer, and a convolutional layer; MDF blocks are used to extract and fuse chaotic domain features and frequency domain features from the feature sequence in the preliminary features from the two dimensions of height and width.

4. The method for identifying body constitution and organs according to claim 1 or 3, characterized in that, The steps for feature extraction in chaotic domains are as follows: The input features are subjected to chaotic transformation sequentially through linear transformation, normalization layer and activation function; The transformed features are mapped using Logistic regression to obtain chaotic domain features; the mapping method is as follows: F chaos =μ·Φ(1-Φ) In the formula, Φ represents the chaotic transformation characteristic, Φ chaos denoted by , which represents the characteristics of the chaotic domain, and μ represents the chaotic parameter.

5. The method for identifying body constitution and organs according to claim 1 or 3, characterized in that, The formula for frequency domain feature extraction is: In the formula, x represents the input feature, DFT[·] represents the two-dimensional discrete Fourier transform function, iDFT[·] represents the two-dimensional inverse discrete Fourier transform function, and H d This represents the learnable weights.

6. The method for identifying body constitution and organs according to claim 1 or 3, characterized in that, Features from the chaotic domain and frequency domain are concatenated, and feature fusion is achieved through a linear layer.

7. The method for identifying body constitution and organs according to claim 6, characterized in that, During feature fusion, residual connections are made with the features of the input tongue image.

8. The method for identifying body constitution and organs according to claim 6, characterized in that, After acquiring the tongue image, it is divided into blocks; preliminary features of each block are extracted to form a corresponding feature sequence; And inputting multiple feature sequences into a multi-domain fusion MLP model; The feature fusion also includes: Aggregate the fused features of each sub-feature in the feature sequence; The aggregated features of each feature sequence are aggregated again to obtain the fused features of the tongue image.

9. The method for identifying body constitution and organs according to claim 1, characterized in that, Based on key location feature loss, the multi-domain fusion MLP model is trained and optimized: L total =λ·L BCE +(1-λ)·(L con +L loc ) Where, L total L represents the loss function for training. BCE This represents the binary cross-entropy loss, where λ is the smoothing coefficient, and L... con Key location loss of tongue image representing body type, L loc Key location loss of tongue images representing organ categories.

Images