A method and system for encoding visceral organ attributes by fusing multimodal features

Abstract

This invention discloses a method and system for encoding visceral organ attributes by fusing multimodal features, including a data acquisition module, a data processing module, a multimodal fusion feature encoding model, and a model construction and training module. The method involves acquiring tongue images and patient voices, labeling them to obtain corresponding visceral organ attribute tags, processing the data separately, and using a deep neural network model to extract individual features from the tongue image modality and voice modality respectively. Constrained by the consistency and complementarity of the representation space, the individual features of the tongue image data and voice data are fused into modal features. Supervised learning is then performed using visceral organ tags and organ attribute tags to embed prior guidance knowledge of visceral organ attributes, resulting in a multimodal fusion feature encoding model. After processing the acquired tongue images and patient voices, the corresponding visceral organ attribute tags are obtained using the multimodal fusion feature encoding model, improving the accuracy and objectivity of visceral organ attribute encoding.

Description

Technical Field

[0001] This invention relates to the field of machine learning technology, and more specifically to a method and system for encoding visceral organ attributes by fusing multimodal features. Background Technology

[0002] In recent years, artificial intelligence has developed rapidly, especially deep neural network machine learning technology based on big data, which has achieved great development and widespread application. Western medicine has already widely applied artificial intelligence and achieved a series of breakthroughs, such as deep learning-based image diagnosis and automated medical record processing. The application of these technologies has greatly improved the efficiency and accuracy of medical diagnosis, and the application of deep learning technology in medical diagnosis has become a trend.

[0003] Currently, the application of Traditional Chinese Medicine (TCM) in the field of artificial intelligence is receiving increasing attention. Unlike Western medicine, TCM diagnosis is based on a holistic view and dialectical thinking, emphasizing syndrome differentiation and treatment. TCM diagnostic methods include the four diagnostic methods of observation, auscultation and olfaction, inquiry, and palpation. By comprehensively judging the patient's symptoms such as complexion, eyes, tongue, voice, and pulse, the patient's condition is determined. Among them, tongue appearance and voice are the contents of observation and auscultation and olfaction in TCM diagnostic methods, and also serve as the basis for internal organs and their properties.

[0004] However, current research on the encoding of visceral organs and their attributes using deep learning-based multimodal methods of tongue image and speech still has many shortcomings. On the one hand, most current diagnostic models are based on machine learning for encoding visceral organs and their attributes. These algorithms cannot consider the inherent attributes of traditional Chinese medicine (TCM) characteristics. TCM's classification and diagnostic criteria for organs differ from those of Western medicine, requiring appropriate adjustments and optimizations to the data and algorithms. For example, factors such as tongue shape, color, and coating texture need to be considered, as well as the tone, timbre, audio time domain, and frequency domain of the voice. These features cannot be processed by traditional machine learning algorithms. On the other hand, according to the principles of TCM thinking and syndrome differentiation and treatment, multimodal diagnosis is a unique diagnostic method in TCM. However, there is currently no research on multimodal diagnosis of visceral organs and their attributes. Although single-modal encoding and diagnostic models have good results, there is still a certain degree of subjectivity and inconsistency in the treatment process. It is necessary to combine different modalities and conduct comprehensive diagnosis from different perspectives.

[0005] Therefore, encoding the attributes of internal organs by combining tongue image data and voice data based on deep learning technology from the perspective of multimodal diagnosis is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0006] In view of this, the present invention provides a method and system for encoding visceral organ attributes by fusing multimodal features to solve the problems mentioned in the background art.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] An visceral organ attribute encoding method that integrates multimodal features includes the following steps:

[0009] S1. Acquire tongue images and patient voices, and label the internal organ attributes corresponding to the acquired tongue images and voices. The labels include internal organ category labels and organ attribute labels corresponding to each internal organ.

[0010] S2. Process the tongue image data and patient voice data separately to obtain the processed batch tongue image data and voice data spectrograms;

[0011] S3. Using the processed batch of tongue images as input image data and the converted batch of spectrograms as input sound data, a deep neural network model is used to extract the individual features of the tongue image modality and the individual features of the sound modality, respectively.

[0012] S4. Constrained by the consistency and complementarity of the representation space, the individual features of tongue image data and the individual features of sound data are fused into modal features. Supervised learning is then performed using visceral organ labels and organ attribute labels to embed prior knowledge of visceral organ attributes, thereby obtaining a multimodal fusion feature encoding model of visceral organ attributes that embeds prior knowledge.

[0013] Preferably, the visceral organ attribute encoding method that integrates multimodal features further includes:

[0014] S5. Collect tongue images and patient voices, process the tongue image data and patient voice data separately to obtain batch tongue image data and voice data spectrograms, input them into a multimodal fusion feature coding model that embeds prior knowledge of visceral organ attributes, and obtain visceral organ attribute labels corresponding to the tongue image and voice.

[0015] Preferably, step S2 includes the following:

[0016] The target detection model is used to detect tongue coating and crop the target region in the tongue image data. The image is expanded by bilinear interpolation and randomly cropped according to the original size to obtain an output image copy of the same size as the original. The output image copy is horizontally flipped and the three basic color channels of the image (red, green and blue) are normalized respectively.

[0017] Audio noise reduction is performed using a speech denoising model. Silent frames are removed using audio and music signal processing tools. Sound segments are randomly extracted and pre-emphasized, framed, and windowed. The time-domain sound signal is converted into a spectrogram through time-frequency transformation.

[0018] Preferably, the deep neural network models in S3 include convolutional neural network ensemble MLP models and recurrent neural network ensemble MLP models;

[0019] Using batches of processed tongue images as input image data, a convolutional neural network ensemble MLP model is used to extract individual features from the image data;

[0020] Using batch spectrograms as input sound data, a recurrent neural network ensemble MLP model is used to extract the individual features of the sound data.

[0021] Preferably, the convolutional neural network ensemble MLP model includes several convolutional layers, normalization layers, downsampling layers, and fully connected layers;

[0022] The combined recurrent neural network (RNN) MLP model includes several recurrent units, normalization layers, downsampling layers, and fully connected layers.

[0023] Preferably, the specific content of step S4 is as follows:

[0024] S41. Calculate the Euclidean distance between the individual features of the tongue image modality and the individual features of the voice modality in Euclidean space; calculate the hyperbolic distance between the individual features of the tongue image feature modality and the individual features of the voice modality in hyperbolic space; calculate the cosine similarity between the Euclidean distance and the hyperbolic distance, which represents the consistency of the space.

[0025] S42. Using a cross-modal bridging fusion strategy, the individual features of the tongue image modality and the individual features of the voice modality are fused, and the Sigmoid activation function is used to output the visceral organ attribute features.

[0026] S43. Map the output visceral organ attribute features to Euclidean space and hyperbolic space respectively, and calculate the cross-entropy loss in their respective representation spaces, i.e. the complementarity of the representation spaces.

[0027] S44. By combining the consistency and complementarity of the representation space, a loss function for the attributes of internal organs is obtained. The loss function is updated by training parameters of the model after multiple iterations of training, and a multimodal fusion feature encoding model is obtained.

[0028] Preferably, the specific content representing spatial consistency in S41 is as follows:

[0029]

[0030] in, This represents the mapping of tongue-like modal characteristics in European space. This represents the mapping of the individual characteristics of sound modalities in Euclidean space. Mapping tongue image modal individual characteristics in hyperbolic space. The mapping of individual characteristics of sound modalities in hyperbolic space;

[0031] d e Euclidean distance metric:

[0032]

[0033] For hyperbolic space distance measurement:

[0034]

[0035] Where c is the spatial curvature constant, c<0;

[0036] d1 takes the cosine distance loss:

[0037]

[0038] Preferably, the specific content representing the complementarity of spaces is as follows:

[0039] The Euclidean spatial distance similarity metric is:

[0040]

[0041] The similarity metric for hyperbolic space structures is:

[0042]

[0043] d2 and d3 respectively take the cross-entropy loss d ce Hyperbolic spatial distance

[0044]

[0045]

[0046] Where c is the spatial curvature constant, c<0.

[0047] Preferably, the specific content of S44 is as follows:

[0048] The consistency constraints of modal individual characteristics in Euclidean and hyperbolic spaces are as follows:

[0049]

[0050] The complementarity constraints of modal individual characteristics in Euclidean and hyperbolic spaces are as follows:

[0051]

[0052] The loss function of the multimodal fusion feature encoding model is:

[0053]

[0054] Where W represents the weight of each sub-item, specifically, W ce for The weight, W consis for The weight, W compl for The weight.

[0055] An visceral organ attribute coding system that integrates multimodal features includes a data acquisition module, a data processing module, a multimodal fusion feature coding model, and a model building and training module;

[0056] The model building and training module includes a labeling unit, a feature extraction unit, a modal feature fusion unit, and a supervised learning unit;

[0057] The data acquisition module is used to acquire images of the tongue and the patient's voice;

[0058] The data processing module is used to process the tongue image data and patient voice data separately to obtain processed batch tongue image data and voice data spectrograms.

[0059] A multimodal fusion feature encoding model is used to obtain the visceral organ attribute labels corresponding to the tongue image and sound based on the processed batch tongue image data and sound data spectrogram;

[0060] The model building and training module is used to build and train a multimodal fusion feature encoding model.

[0061] The annotation unit is used to annotate the internal organ attribute labels corresponding to the obtained tongue image and sound. The labels include internal organ category labels and organ attribute labels corresponding to each internal organ.

[0062] The feature extraction unit is used to extract the individual features of the tongue image modality and the individual features of the sound modality using a deep neural network model, taking the processed batch of tongue images as input image data and the converted batch of spectrograms as input sound data.

[0063] The modal feature fusion unit is used to fuse modal features of tongue image data and sound data with constraints of consistency and complementarity of representation space.

[0064] The supervised learning unit utilizes visceral organ labels and organ attribute labels for supervised learning, embedding prior guidance knowledge of visceral organ attributes to obtain a multimodal fusion feature encoding model of visceral organ attributes that incorporates prior knowledge.

[0065] As can be seen from the above technical solutions, compared with the prior art, the present invention discloses a method and system for encoding visceral organ attributes by fusing multimodal features. By combining tongue image and sound data and using multimodal fusion, it can more comprehensively understand the information contained in each modality from different perspectives and modalities, thereby mining the potential patterns and rules behind the data and improving the classification performance. It proposes the relationship between the consistency and complementarity of the representation space and applies it to the learning of multimodal models. By utilizing the consistent and complementary components between different modal data, it improves the ability of multimodal feature fusion. Attached Figure Description

[0066] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0067] Figure 1 The attached figure is a schematic diagram of the model structure of the multimodal visceral organ attribute encoding method provided by the present invention;

[0068] Figure 2 The attached figure is a schematic diagram of the neural network ensemble MLP model structure provided by the present invention;

[0069] Figure 3 The attached figure is a schematic diagram of the structural parameters of the multimodal fusion feature coding model provided by the present invention. Detailed Implementation

[0070] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0071] This invention discloses a method for encoding visceral organ attributes by fusing multimodal features, comprising the following steps:

[0072] S1. Acquire tongue images and patient voices, and label the internal organ attributes corresponding to the acquired tongue images and voices. The labels include internal organ category labels and organ attribute labels corresponding to each internal organ.

[0073] S2. Process the tongue image data and patient voice data separately to obtain the processed batch tongue image data and voice data spectrograms;

[0074] S3. Using the processed batch of tongue images as input image data and the converted batch of spectrograms as input sound data, a deep neural network model is used to extract the individual features of the tongue image modality and the individual features of the sound modality, respectively.

[0075] S4. Constrained by the consistency and complementarity of the representation space, multimodal feature fusion is performed on the individual features of tongue image data and the individual features of sound data. Supervised learning is then performed using visceral organ labels and organ attribute labels to embed prior guidance knowledge of visceral organ attributes, thereby obtaining a multimodal fusion feature encoding model of visceral organ attributes embedded with prior knowledge.

[0076] In this embodiment, the visceral organ category labels include large intestine, gallbladder, lung, liver, bladder, spleen, kidney, stomach, small intestine, heart, and unknown; the organ attribute labels include qi deficiency, blood deficiency, yin deficiency, yang deficiency, qi stagnation, blood stasis, phlegm, wind, heat, cold, dryness, dampness, and unknown.

[0077] To further implement the above technical solution, a method for encoding visceral organ attributes that integrates multimodal features also includes:

[0078] S5. Collect tongue images and patient voices, process the tongue image data and patient voice data separately to obtain batch tongue image data and voice data spectrograms, input them into a multimodal fusion feature coding model that embeds prior knowledge of visceral organ attributes, and obtain visceral organ attribute labels corresponding to the tongue image and voice.

[0079] To further implement the above technical solution, step S2 includes the following:

[0080] The Faster R-CNN object detection model was used to detect tongue coating. The target region was then cropped to form an image of size 224×224. The image was then scaled to 256×256 using bilinear interpolation. The image was then randomly cropped to the original size of 224×224 to obtain an output image copy of the same size. The output image copy was then horizontally flipped with a probability of 0.5. The three basic color channels of the image (red, green, and blue) were then normalized.

[0081] The PHASEN speech denoising model was used for audio noise reduction. The audio and music signal processing tool Librosa was used to remove silent frames. Audio segments with a length of not less than 10 seconds and not more than 90 seconds were randomly selected and pre-emphasized, framed and windowed. The time-domain audio signal was converted into a spectrum graph through time-frequency transformation.

[0082] To further implement the above technical solutions, the deep neural network models in S3 include convolutional neural network ensemble MLP models and recurrent neural network ensemble MLP models;

[0083] Using batches of processed tongue images as input image data, a convolutional neural network ensemble MLP model is used to extract individual features from the image data:

[0084] Z t =MLP(CNN(X) t ))

[0085] Using batch spectrograms as input sound data, a recurrent neural network ensemble MLP model is used to extract individual features from the sound data:

[0086] Z s =MLP(RNN(X) s ))

[0087] Where, X = {X} t ,X s Let} be the input sample set, X t Tongue image, X s For sound, Z t To extract individual characteristics from tongue images, Z s To extract the unique characteristics of the voice.

[0088] To further implement the above technical solutions, the convolutional neural network ensemble MLP model includes several convolutional layers, normalization layers, downsampling layers, and fully connected layers;

[0089] The combined recurrent neural network (RNN) MLP model includes several recurrent units, normalization layers, downsampling layers, and fully connected layers.

[0090] In this embodiment, the MLP-like system receives input in three dimensions, consisting of three branches. Each branch is responsible for encoding information along the height, width, and channel dimensions. After encoding each branch, X H X W and X C Specifically:

[0091] The channel information is encoded using a weight of A fully connected layer (FC) is used to project the input X to obtain X. C ;

[0092] Divide the given input into S segments along the channel dimension to obtain For each segment Perform a height-channel permutation operation to obtain Then, connect along the channel dimension as the output of the permutation operation. Next, connect a weight of A fully connected (FC) layer is used to mix the height information. Finally, a height-channel permutation operation is performed to restore the original dimensional information, outputting X. H Similarly, in order to obtain spatial information along the width for encoding, in the second branch, we perform the same operation as above to permutate the width and channel dimensions of X and generate X. W ;

[0093] Using the three weights generated by separate attention (A) H A W A C ) and the three latent variables (X) after encoding H ,X W ,X C Perform a weighted summation to obtain the output.

[0094] A = FC(X) H +X W +X C )

[0095] [A H A W A C = softmax(A)

[0096]

[0097] To further implement the above technical solution, the specific content of step S4 is as follows:

[0098] S41. Calculate the Euclidean distance between the individual features of the tongue image modality and the individual features of the voice modality in Euclidean space; calculate the hyperbolic distance between the individual features of the tongue image feature modality and the individual features of the voice modality in hyperbolic space; calculate the cosine similarity between the Euclidean distance and the hyperbolic distance, which represents the consistency of the space.

[0099] S42. Using a cross-modal bridging fusion strategy, the individual features of the tongue image modality and the individual features of the voice modality are fused, and the Sigmoid activation function is used to output the visceral organ attribute features.

[0100] S43. Map the output visceral organ attribute features to Euclidean space and hyperbolic space respectively, and calculate the cross-entropy loss in their respective representation spaces, i.e. the complementarity of the representation spaces.

[0101] S44. By combining the consistency and complementarity of the representation space, a loss function for the attributes of internal organs is obtained. The loss function is updated by training parameters of the model after multiple iterations of training, and a multimodal fusion feature encoding model is obtained.

[0102] To further implement the above technical solution, the specific content of the consistency representation space in S41 is as follows:

[0103]

[0104] in, This represents the mapping of tongue-like modal characteristics in European space. This represents the mapping of the individual characteristics of sound modalities in Euclidean space. Mapping tongue image modal individual characteristics in hyperbolic space. The mapping of individual characteristics of sound modalities in hyperbolic space;

[0105] d e Euclidean distance metric:

[0106]

[0107] For hyperbolic space distance measurement:

[0108]

[0109] Where c is the spatial curvature constant, c<0;

[0110] d1 takes the cosine distance loss:

[0111]

[0112] In this embodiment, the mapping relationship between the tongue image in the two spaces is as follows:

[0113]

[0114]

[0115] The mapping relationship between sound in two spaces is as follows:

[0116]

[0117]

[0118] in, For Möbius addition, c < 0. In particular, when c = 0, it degenerates into addition in Euclidean space.

[0119] To further implement the above technical solution, the specific content representing the complementarity of spaces in S43 is as follows:

[0120] The Euclidean spatial distance similarity metric is:

[0121]

[0122] The similarity metric for hyperbolic space structures is:

[0123]

[0124] in, It is a mapping of visceral organ attribute prediction in Euclidean space, Y e It is a mapping of internal organ attribute labels in Euclidean space; It is a mapping of visceral organ attribute prediction in hyperbolic space, Y h It is a mapping of visceral organ attribute labels in hyperbolic space;

[0125] d2 takes the cross-entropy loss d ce :

[0126]

[0127] d3 represents the hyperbolic spatial distance

[0128]

[0129] Where c is the spatial curvature constant, c<0.

[0130] To further implement the above technical solution, the specific content of S44 is as follows:

[0131] The consistency constraints of modal individual characteristics in Euclidean and hyperbolic spaces are as follows:

[0132]

[0133] The complementarity constraints of modal individual characteristics in Euclidean and hyperbolic spaces are as follows:

[0134]

[0135] The loss function of the multimodal fusion feature encoding model is:

[0136]

[0137] Where W represents the weight of each sub-item. Specifically, W... ce for The weight, W consis for The weight, W compl for The weight.

[0138] In practical applications, the deep learning framework PyTorch and the model library timm were used. All experiments were run on a server equipped with two NVIDIA GeForce GTX 1080 GPUs, each with approximately 12GB of memory. The operating system was Ubuntu 16.04. The Stochastic Gradient Descent (SGD) algorithm was used for training, with the following parameters: weight decay of 5e-4, momentum of 0.9, and batch size of 64. The total number of training epochs was 100, with an initial learning rate of 0.01. From the 60th epoch, cosine annealing was used to decay the learning rate to the minimum, which was set to 2e-4.

[0139] An visceral organ attribute coding system that integrates multimodal features includes a data acquisition module, a data processing module, a multimodal fusion feature coding model, and a model building and training module;

[0140] The model building and training module includes a labeling unit, a feature extraction unit, a modal feature fusion unit, and a supervised learning unit;

[0141] The data acquisition module is used to acquire images of the tongue and the patient's voice;

[0142] The data processing module is used to process the tongue image data and patient voice data separately to obtain processed batch tongue image data and voice data spectrograms.

[0143] A multimodal fusion feature encoding model is used to obtain the visceral organ attribute labels corresponding to the tongue image and sound based on the processed batch tongue image data and sound data spectrogram;

[0144] The model building and training module is used to build and train a multimodal fusion feature encoding model.

[0145] The annotation unit is used to annotate the internal organ attribute labels corresponding to the obtained tongue image and sound. The labels include internal organ category labels and organ attribute labels corresponding to each internal organ.

[0146] The feature extraction unit is used to extract the individual features of the tongue image modality and the individual features of the sound modality using a deep neural network model, taking the processed batch of tongue images as input image data and the converted batch of spectrograms as input sound data.

[0147] The modal feature fusion unit is used to fuse modal features of tongue image data and sound data with constraints of consistency and complementarity of representation space.

[0148] The supervised learning unit utilizes visceral organ labels and organ attribute labels for supervised learning, embedding prior guidance knowledge of visceral organ attributes to obtain a multimodal fusion feature encoding model of visceral organ attributes that incorporates prior knowledge.

[0149] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0150] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for encoding visceral organ attributes by fusing multimodal features, characterized in that, Includes the following steps: S1. Collect tongue images and patient voices, and label the internal organ attributes corresponding to the obtained tongue images and voices. The labels include internal organ category labels and organ attribute labels corresponding to each internal organ. S2. Process the tongue image data and patient voice data separately to obtain the processed batch tongue image data and voice data spectrograms; S3. Using the processed batch of tongue images as input image data and the converted batch of spectrograms as input sound data, a deep neural network model is used to extract the individual features of the tongue image modality and the individual features of the sound modality, respectively. S4. Constrained by the consistency and complementarity of the representation space, multimodal feature fusion is performed on the individual features of the tongue image modality and the individual features of the voice modality. Supervised learning is performed using visceral organ labels and organ attribute labels to embed prior guidance knowledge of visceral organ attributes, thereby obtaining a multimodal fusion feature encoding model of visceral organ attributes embedded with prior knowledge. The specific content of step S4 is as follows: S41. Calculate the Euclidean distance between the individual features of the tongue image mode and the individual features of the sound mode in Euclidean space; Calculate the hyperbolic distance between the individual features of the tongue image modality and the individual features of the sound modality in hyperbolic space; calculate the cosine similarity between the Euclidean distance and the hyperbolic distance, which represents the consistency of the space; S42. Using a cross-modal bridging fusion strategy, the individual features of the tongue image modality and the individual features of the voice modality are fused, and the Sigmoid activation function is used to output the visceral organ attribute features. S43. Map the output visceral organ attribute features to Euclidean space and hyperbolic space respectively, and calculate the cross-entropy loss in their respective representation spaces, i.e., the complementarity of the representation spaces. S44. By combining the consistency and complementarity of the representation space, a loss function for the attributes of internal organs is obtained. The loss function is updated by training parameters of the model after multiple iterations of training, and a multimodal fusion feature encoding model is obtained.

2. The method for encoding visceral organ attributes by fusing multimodal features according to claim 1, characterized in that, Also includes: S5. Collect tongue images and patient voices, process the tongue image data and patient voice data separately to obtain the processed batch tongue image data and voice data spectrograms, input them into a multimodal fusion feature coding model that embeds prior knowledge of visceral organ attributes, and obtain the visceral organ attribute labels corresponding to the tongue image and voice.

3. The method for encoding visceral organ attributes by fusing multimodal features according to claim 1, characterized in that, The specific content of step S2 includes: The target detection model is used to detect tongue coating and crop target regions in tongue image data. The image is expanded by bilinear interpolation and randomly cropped according to the original size to obtain an output image copy of the same size as the original. The output image copy is horizontally flipped and the three basic color channels of red, green and blue are normalized respectively. Audio noise reduction is performed using a speech denoising model. Silent frames are removed using audio and music signal processing tools. Sound segments are randomly extracted and pre-emphasized, framed, and windowed. The time-domain sound signal is converted into a spectrogram through time-frequency transformation.

4. The method for encoding visceral organ attributes by fusing multimodal features according to claim 1, characterized in that, The deep neural network models in S3 include convolutional neural network ensemble MLP models and recurrent neural network ensemble MLP models; Using batches of processed tongue images as input image data, a convolutional neural network ensemble MLP model is used to extract individual features from the image data; Using batch spectrograms as input sound data, a recurrent neural network ensemble MLP model is used to extract the individual features of the sound data.

5. The method for encoding visceral organ attributes by fusing multimodal features according to claim 4, characterized in that, Convolutional neural network ensemble MLP models include several convolutional layers, normalization layers, downsampling layers, and fully connected layers; The combined recurrent neural network (RNN) MLP model includes several recurrent units, normalization layers, downsampling layers, and fully connected layers.

6. The method for encoding visceral organ attributes by fusing multimodal features according to claim 1, characterized in that, The specific content representing the consistency of space in S41 is as follows: ； in, This represents the mapping of the individual characteristics of the tongue image modality in Euclidean space. This refers to the mapping of the individual characteristics of sound modes in Euclidean space. This represents the mapping of individual characteristics of the tongue image modality in hyperbolic space. The mapping of the individual characteristics of sound modes in hyperbolic space; Euclidean distance metric: ； For hyperbolic space distance measurement: ； in, Let be the space curvature constant. ; Cosine distance loss: 。 7. The method for encoding visceral organ attributes by fusing multimodal features according to claim 6, characterized in that, The specific content representing the complementarity of spaces is as follows: The Euclidean spatial distance similarity metric is: ； The similarity metric for hyperbolic space structures is: ； in, It is a mapping of the predicted attributes of internal organs in Euclidean space. It is a mapping of internal organ attribute labels in Euclidean space; It is a mapping of visceral organ attribute prediction in hyperbolic space. It is a mapping of visceral organ attribute labels in hyperbolic space; Take cross-entropy loss , Take hyperbolic space distance : ； ； in, Let be the space curvature constant. .

8. The method for encoding visceral organ attributes by fusing multimodal features according to claim 7, characterized in that, The specific content of S44 is as follows: The consistency constraints of modal individual characteristics in Euclidean and hyperbolic spaces are as follows: ； The complementarity constraints of modal individual characteristics in Euclidean and hyperbolic spaces are as follows: ； The loss function of the multimodal fusion feature encoding model is: ； Where W represents the weight of each sub-item, specifically, for The weight, for The weight, for The weight.

9. A visceral organ attribute coding system integrating multimodal features, based on the visceral organ attribute coding method integrating multimodal features as described in any one of claims 1-8, characterized in that, It includes a data acquisition module, a data processing module, a multimodal fusion feature encoding model, and a model building and training module; The model building and training module includes a labeling unit, a feature extraction unit, a modal feature fusion unit, and a supervised learning unit; The data acquisition module is used to collect tongue images and patient voices; The data processing module is used to process the tongue image data and patient voice data separately to obtain the processed batch tongue image data and voice data spectrograms. A multimodal fusion feature encoding model is used to obtain the visceral organ attribute labels corresponding to the tongue image and sound based on the processed batch tongue image data and sound data spectrogram; The model building and training module is used to build and train a multimodal fusion feature encoding model. The annotation unit is used to annotate the internal organ attribute labels corresponding to the obtained tongue image and sound. The labels include internal organ category labels and organ attribute labels corresponding to each internal organ. The feature extraction unit is used to extract the individual features of the tongue image modality and the individual features of the sound modality using a deep neural network model, with the processed batch of tongue images as input image data and the converted batch of spectrograms as input sound data. The modal feature fusion unit is used to fuse the individual features of the tongue image modality and the individual features of the sound modality, constrained by the consistency and complementarity of the representation space. The supervised learning unit utilizes visceral organ labels and organ attribute labels for supervised learning, embedding prior guidance knowledge of visceral organ attributes to obtain a multimodal fusion feature encoding model of visceral organ attributes that incorporates prior knowledge.

Images