A Fusion Method for Multimodal Diagnosis Information in Traditional Chinese Medicine and Related Devices

Abstract

The present invention discloses a method for fusing multi-modal diagnosis and treatment information in traditional Chinese medicine and related devices. The method includes: obtaining multi-modal diagnosis and treatment data, and decomposing each modal diagnosis and treatment data into common information and unique information within each modality; based on the disentanglement strategy, regarding the common information between multiple modalities and the unique information of each modality as different categories, and fully learning the unique information of different modalities through a classifier in a new projection space; concatenating the learned common information and unique information to obtain fused features, and through classification and multiple optimization iterations, enhancing the representation effect of the fused features on diseases. The present invention decomposes multi-modal diagnosis and treatment data into common information and unique information, designs a constraint function and a classification model through the disentanglement strategy, enhances the unique information and the common information, concatenates the common information and the unique information, and further enhances the unique information and the common information through classification and multiple optimization iterations, achieving a good multi-modal feature representation effect.

Description

Technical Field

[0001] The present invention relates to the technical field of computer information science and technology, and particularly to a method, system, terminal and computer-readable storage medium for fusing multi-modal diagnosis information of traditional Chinese medicine. Background Art

[0002] A modality is a way in which something occurs or exists. Multi-modal refers to the combination of various forms of two or more modalities. In the medical field, multi-modal mostly refers to representing the diseases of the same patient from different perspectives. For example, in traditional Chinese medicine diagnosis, the pulse condition, tongue image, sublingual vein of the same patient, etc.; in Western medicine diagnosis, CT image information, X-ray image, magnetic resonance image, etc. The above information reflects different pathological information of the same patient from different perspectives, and thus can be considered as medical multi-modal data. The reason for fusing modalities is that the same data source has different manifestations in different modalities, just as the information obtained from looking at the same thing from different perspectives is also different. Therefore, multi-modal fusion is needed to organize and combine the obtained data, so as to better depict the original situation of the data source. Since different modal data come from different acquisition forms of the same data source, there are often some cross and complementary phenomena in multi-modal data. If multi-modal information can be reasonably processed, rich feature information can be obtained. Generally speaking, the remarkable characteristics of multi-modal are redundancy and complementarity.

[0003] To achieve the fusion of multi-modal medical data, some studies combine multi-modal features into a single feature vector to better represent diseases; there are also studies that use composite kernel learning to fuse the pulse signals of traditional Chinese medicine in different modalities; for example, the multi-feature fusion algorithm based on Canonical Correlation Analysis (CCA) is also a common multi-modal medical data fusion strategy, and this method can improve the representation performance of multi-modal medical data. In addition, Multiple Kernel Learning (MKL) can also effectively combine multi-modal medical data. This method compresses multiple modalities and can cover information from multiple perspectives.

[0004] Although there are various methods to achieve multimodal traditional Chinese medicine (TCM) data fusion, most methods fail to consider the complementarity of different features and diverse information, resulting in a decline in diagnostic performance. Some studies directly concatenate different features into a single vector and use it for analysis, which contains a lot of redundant information. In addition, the CCA-based algorithm simply reduces the redundancy between different modal information, but also loses the key diagnostic information for disease analysis; while MKL only focuses on the unique information between different modalities and has some deficiencies in reducing redundancy. Therefore, when performing diagnostic analysis based on multimodal diagnosis and treatment data, it is very necessary to fuse multiple pulse features to obtain better results.

[0005] Therefore, the existing technology still needs to be improved and developed. Summary of the Invention

[0006] The main objective of the present invention is to provide a method, system, terminal, and computer-readable storage medium for fusing multimodal diagnostic information in traditional Chinese medicine, aiming to solve the problem in the existing technology that redundant information cannot be effectively reduced and key diagnostic information is not lost when performing multimodal medical data fusion.

[0007] To achieve the above objective, the present invention provides a method for fusing multimodal diagnostic information in traditional Chinese medicine, and the method for fusing multimodal diagnostic information in traditional Chinese medicine includes the following steps:

[0008] Obtain multiple modal diagnosis and treatment data, and decompose each modal diagnosis and treatment data into common information and unique information within each modality;

[0009] Based on the disentanglement strategy, regard the common information between multiple modalities and the unique information of each modality as different categories, and fully learn the unique information of different modalities through a classifier in a new projection space;

[0010] Concatenate the learned common information and unique information to obtain a fused feature, and through classification and multiple optimization iterations, enhance the representation effect of the fused feature on the disease.

[0011] Optionally, in the method for fusing multimodal diagnostic information in traditional Chinese medicine, the step of obtaining multiple modal diagnosis and treatment data and decomposing each modal diagnosis and treatment data into common information and unique information within each modality specifically includes:

[0012] If there are v modalities, then:

[0013]

[0014] where X v represents the v-th modality of a batch of samples; d v is the dimension of the v-th modality; n is the number of samples for training; Denote the data of the v-th modality of the i-th training sample; V represents the number of modalities; Denote the vector space;

[0015] Decompose the diagnosis and treatment data of each modality into common information and unique information

[0016] Common information: Denote the part that is similar and common between the v-th modality and other modalities;

[0017] Unique information: Denote the part that the v-th modality has and other modalities do not have;

[0018] where q represents a dimension after the sparse dictionary.

[0019] Optionally, for the method for fusing multi-modal diagnosis and treatment information of traditional Chinese medicine, where the obtaining of multi-modal diagnosis and treatment data and the decomposition of the diagnosis and treatment data of each modality into common information and unique information within each modality are followed by:

[0020] Project X v as:

[0021]

[0022] where M v denotes the sparse dictionary; ||·|| F is the F-norm.

[0023] Optionally, for the method for fusing multi-modal diagnosis and treatment information of traditional Chinese medicine, where the projection of X v is followed by:

[0024] Use to represent Then the projection function of the modality information is optimized as:

[0025]

[0026] where L pair denotes the projection function; denotes the sparse matrix of the unique information after disassembling the v-th modality; denotes the sparse matrix of the common information after disassembling the v-th modality.

[0027] Optionally, for the method for fusing multi-modal diagnosis and treatment information of traditional Chinese medicine, if the common information between multiple modalities is Z, then the common information of the multi-modalities is represented as:

[0028]

[0029] Optionally, in the method for fusing multi-modal diagnosis information in traditional Chinese medicine, based on the disentanglement strategy, the common information between multiple modalities and the unique information of each modality are regarded as different categories, and in the new projection space, a classifier is used to fully learn the unique information of different modalities, which specifically includes:

[0030] Regard the common information Z between multiple modalities and the unique information of each modality as different categories, and the label of the category is L f , in the new projection space, use a classifier P to fully learn the unique information of different modalities, and the learning strategy is:

[0031]

[0032] Optionally, in the method for fusing multi-modal diagnosis information in traditional Chinese medicine, the common information and unique information obtained by learning are concatenated to obtain a fusion feature, and through classification and multiple optimization iterations, the representation effect of the fusion feature on the disease is enhanced, which specifically includes:

[0033] Concatenate the common information and unique information obtained by learning, and sequentially combine them into a column vector or a row vector. By designing a classifier U, establish a mapping function between the fusion feature and the disease category Y:

[0034]

[0035] Combined with Equation (1), Equation (2), Equation (3), Equation (4) and Equation (5), the final multi-modal medical diagnosis data fusion function is expressed as:

[0036]

[0037] Among them, α, β and δ represent dynamically adjustable weight coefficients.

[0038] In addition, to achieve the above object, the present invention also provides a system for fusing multi-modal diagnosis information in traditional Chinese medicine, wherein the system for fusing multi-modal diagnosis information in traditional Chinese medicine includes:

[0039] A data decomposition module, configured to obtain multi-modal diagnosis data, and decompose each modal diagnosis data into common information and unique information within each modality;

[0040] An information learning module, configured to, based on the disentanglement strategy, regard the common information between multiple modalities and the unique information of each modality as different categories, and in the new projection space, use a classifier to fully learn the unique information of different modalities;

[0041] A feature fusion module, configured to concatenate the common information and unique information obtained by learning to obtain a fusion feature, and through classification and multiple optimization iterations, enhance the representation effect of the fusion feature on the disease.

[0042] In addition, to achieve the above object, the present invention further provides a terminal, wherein the terminal includes: a memory, a processor, and a fusion program of traditional Chinese medicine multi-modal examination information stored on the memory and executable on the processor. When the fusion program of traditional Chinese medicine multi-modal examination information is executed by the processor, the steps of the above-mentioned fusion method of traditional Chinese medicine multi-modal examination information are implemented.

[0043] In addition, to achieve the above object, the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a fusion program of traditional Chinese medicine multi-modal examination information. When the fusion program of traditional Chinese medicine multi-modal examination information is executed by a processor, the steps of the above-mentioned fusion method of traditional Chinese medicine multi-modal examination information are implemented.

[0044] In the present invention, multiple modal diagnosis and treatment data are obtained, and each modal diagnosis and treatment data is decomposed into common information and unique information within each modality; based on a disentanglement strategy, the common information between multiple modalities and the unique information of each modality are regarded as different categories, and in a new projection space, a classifier is used to fully learn the unique information of different modalities; the learned common information and unique information are concatenated to obtain a fusion feature, and through classification and multiple optimization iterations, the representation effect of the fusion feature on the disease is enhanced. The present invention projects multi-modal diagnosis and treatment data into common information and unique information through subspace mapping, designs a constraint function and a classification model through a disentanglement strategy to enhance the unique information and common information, serializes the common information and unique information, and through classification and multiple optimization iterations, further enhances the unique and common information of the information, achieving a good multi-modal feature representation effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 is a flowchart of a preferred embodiment of the fusion method of traditional Chinese medicine multi-modal examination information of the present invention;

[0046] Figure 2 is a schematic flowchart of an optimization algorithm for solving an objective function in a preferred embodiment of the fusion method of traditional Chinese medicine multi-modal examination information of the present invention;

[0047] Figure 3 is a schematic diagram of the principle of a preferred embodiment of the fusion system of traditional Chinese medicine multi-modal examination information of the present invention;

[0048] Figure 4 is a schematic diagram of the operating environment of a preferred embodiment of the terminal of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0049] To make the objectives, technical solutions and advantages of the present invention more clear and definite, the following further describes the present invention in detail with reference to the accompanying drawings and by way of examples. It should be understood that the specific examples described herein are merely for explaining the present invention and are not intended to limit the present invention.

[0050] The method for fusing multi-modal diagnosis information in traditional Chinese medicine according to a preferred embodiment of the present invention is as Figure 1 shown. The method for fusing multi-modal diagnosis information in traditional Chinese medicine includes the following steps:

[0051] Step S10: Obtain multi-modal diagnosis and treatment data, and decompose each modal diagnosis and treatment data into common information and unique information within each modality.

[0052] Specifically, if there are v modalities (for example, v = 4, indicating the existence of 4 modalities), then:

[0053]

[0054] Among them, X v represents the v-th modality of a batch of samples; d v is the dimension of the v-th modality; n is the number of samples for training; represents the data of the v-th modality of the i-th training sample; V represents the number of modalities; represents a vector space. For example, d v ×n represents d v rows and n columns.

[0055] The present invention decomposes each modal diagnosis and treatment data into common information and unique information

[0056] The common information is the similar or common part of each modal data, which represents the public expression of disease information from multiple perspectives and is regarded as similarity. The common information: represents the part similar and common to the v-th modality and other modalities; the unique information represents the information that can be detected by this modality but cannot be reflected by other modalities. The unique information between different modalities is complementary and can be integrated to promote the pattern recognition performance. The unique information: represents the part that the v-th modality has and other modalities do not have; where q represents a dimension after the sparse dictionary.

[0057] For example, for a certain disease with 3 modalities, through the data of tongue coating, pulse, and CT, the common information refers to: a certain disease condition can be reflected in tongue coating, pulse condition, and CT at the same time; the unique information refers to: a certain disease condition can only be reflected in CT or tongue coating.

[0058] Project X v to:

[0059]

[0060] Among them, M v represents a sparse dictionary; ||·|| F is the F-norm.

[0061] Furthermore, use to represent Then the projection function of the modal information is optimized as:

[0062]

[0063] Among them, L pair represents the projection function; represents the sparse matrix of the unique information after the decomposition of the v-th modality; represents the sparse matrix of the common information after the decomposition of the v-th modality.

[0064] The multi-modal information has similarity. If the common information between multiple modalities is Z, the common information of the multi-modal is expressed as:

[0065]

[0066] Step S20: Based on the disentanglement strategy, regard the common information between multiple modalities and the unique information of each modality as different categories, and use a classifier in the new projection space to fully learn the unique information of different modalities.

[0067] Specifically, based on the learning strategy of disentangled unique information and common information, there are huge differences between the common information and the unique information of each modality, and they are far apart in space. Based on this prior information, a decoupling strategy is introduced to realize the learning of unique information and common information.

[0068] Regard the common information Z between multiple modalities and the unique information of each modality as different categories, and the label of the category is L f , and use a classifier P in the new projection space to fully learn the unique information of different modalities. The learning strategy is:

[0069]

[0070] Step S30: Concatenate the learned common information and unique information to obtain a fused feature, and through classification and multiple optimization iterations, enhance the representation effect of the fused feature on the disease.

[0071] Specifically, to achieve a good fusion effect, the learned common information and unique information are concatenated and combined into a column vector or a row vector in sequence. By designing a classifier U (such as a disease classifier), a mapping function between the fusion features and the disease category Y is established:

[0072]

[0073] Combining Equation (1), Equation (2), Equation (3), Equation (4) and Equation (5), the final multi-modal medical examination data fusion function is expressed as:

[0074]

[0075] where α, β and δ represent dynamically adjustable weight coefficients.

[0076] Furthermore, as Figure 2 shown, to solve various table variables ( M v Z, P, R v U), an iterative optimization fusion function optimization algorithm is designed. The input of this optimization solution algorithm is multi-modal data X, L f Y, and the output is M v Z, P, R v U; that is, there are many unknowns in the present invention, so an algorithm needs to be designed to solve these unknowns. At this time, this optimization algorithm is used. The optimization algorithm is iterated many times to find the optimal solution.

[0077] The present invention divides the information of each modality into two parts: common information (Common) and unique information (Specificity) within each modality, and represents them separately. A learning strategy based on disentanglement is designed, which can better learn the common part of multi-modalities and the unique parts of each modality. In addition, to enhance the fusion effect of multi-features, the present invention introduces a disease classifier to further enhance the representation effect of the fusion features on the disease symptoms. The present invention designs an optimization algorithm to obtain the optimal solutions of each mapping matrix through initialization and iteration.

[0078] Furthermore, as Figure 3 shown, based on the above-mentioned fusion method of traditional Chinese medicine multi-modal examination information, the present invention also correspondingly provides a traditional Chinese medicine multi-modal examination information fusion system. Among them, the traditional Chinese medicine multi-modal examination information fusion system includes:

[0079] A data decomposition module 51, configured to obtain multiple modality diagnosis and treatment data, and decompose each modality diagnosis and treatment data into common information and unique information within each modality;

[0080] An information learning module 52, configured to, based on a disentanglement strategy, regard the common information among multiple modalities and the unique information of each modality as different categories, and fully learn the unique information of different modalities through a classifier in a new projection space;

[0081] A feature fusion module 53, configured to concatenate the learned common information and unique information to obtain a fused feature, and enhance the representation effect of the fused feature on the disease condition through classification and multiple optimization iterations.

[0082] Further, as Figure 4 shown, based on the above-mentioned fusion method and system for traditional Chinese medicine multi-modal examination information, the present invention also correspondingly provides a terminal, which includes a processor 10, a memory 20, and a display 30. Figure 4 Only some components of the terminal are shown, but it should be understood that it is not required to implement all the shown components, and more or fewer components can be alternatively implemented.

[0083] The memory 20 may be an internal storage unit of the terminal in some embodiments, such as the hard disk or memory of the terminal. The memory 20 may also be an external storage device of the terminal in other embodiments, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. equipped on the terminal. Further, the memory 20 may also include both the internal storage unit and the external storage device of the terminal. The memory 20 is used to store application software installed on the terminal and various types of data, such as program codes for installing the terminal. The memory 20 may also be used to temporarily store data that has been output or will be output. In one embodiment, a fusion program 40 for traditional Chinese medicine multi-modal examination information is stored on the memory 20, and the fusion program 40 for traditional Chinese medicine multi-modal examination information can be executed by the processor 10, thereby implementing the fusion method for traditional Chinese medicine multi-modal examination information in the present application.

[0084] The processor 10 may be a central processing unit (CPU), a microprocessor, or other data processing chips in some embodiments, and is used to run the program codes stored in the memory 20 or process data, such as executing the fusion method for traditional Chinese medicine multi-modal examination information.

[0085] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch liquid crystal display, an OLED (Organic Light-Emitting Diode) touch device, etc. The display 30 is used to display information of the terminal and to display a visual user interface. The components 10-30 of the terminal communicate with each other through a system bus.

[0086] In one embodiment, when the processor 10 executes the fusion program 40 of the traditional Chinese medicine multi-modal examination information in the memory 20, the following steps are implemented:

[0087] Obtain multi-modal diagnosis and treatment data, and decompose each modal diagnosis and treatment data into common information and unique information within each modality;

[0088] Based on the disentanglement strategy, regard the common information between multiple modalities and the unique information of each modality as different categories, and fully learn the unique information of different modalities through a classifier in a new projection space;

[0089] Concatenate the learned common information and unique information to obtain a fusion feature, and through classification and multiple optimization iterations, enhance the representation effect of the fusion feature on the disease.

[0090] Among them, the obtaining of multi-modal diagnosis and treatment data and the decomposition of each modal diagnosis and treatment data into common information and unique information within each modality specifically include:

[0091] If there are v modalities, then:

[0092]

[0093] Among them, X v represents the v-th modality of a batch of samples; d v is the dimension of the v-th modality; n is the number of samples for training; represents the data of the v-th modality of the i-th training sample; V represents the number of modalities; represents the vector space;

[0094] Decompose each modal diagnosis and treatment data into common information and unique information

[0095] Common information: represents the part that the v-th modality shares in common with other modalities;

[0096] Unique information: represents the part that the v-th modality has and other modalities do not have;

[0097] Among them, q represents a dimension after the sparse dictionary.

[0098] Among them, the steps of obtaining multiple modal diagnosis and treatment data, decomposing each modal diagnosis and treatment data into common information and unique information within each modality, further include:

[0099] Project X v to obtain:

[0100]

[0101] Among them, M v represents the sparse dictionary; ||·|| F is the F-norm.

[0102] Among them, after the step of projecting X v , further include:

[0103] Use to represent Then the projection function of the modal information is optimized to:

[0104]

[0105] Among them, L pair represents the projection function; represents the sparse matrix of the unique information after decomposition of the v-th modality; represents the sparse matrix of the common information after decomposition of the v-th modality.

[0106] Among them, if the common information between multiple modalities is Z, the common information of the multi-modalities is expressed as:

[0107]

[0108] Among them, based on the disentanglement strategy, regarding the common information between multiple modalities and the unique information of each modality as different categories, and fully learning the unique information of different modalities through a classifier in the new projection space, specifically including:

[0109] Regarding the common information Z between multiple modalities and the unique information of each modality as different categories, with the label of the category being L f , and fully learning the unique information of different modalities through the classifier P in the new projection space, and the learning strategy is:

[0110]

[0111] Among them, the steps of concatenating the learned common information and unique information to obtain a fused feature, and enhancing the representation effect of the fused feature on the disease through classification and multiple optimization iterations, specifically include:

[0112] Concatenate the learned shared information and unique information to form a column vector or a row vector in sequence, and establish a mapping function between the fused features and the disease category Y by designing a classifier U:

[0113]

[0114] Combining Equation (1), Equation (2), Equation (3), Equation (4) and Equation (5), the final multi-modal medical examination data fusion function is expressed as:

[0115]

[0116] where α, β and δ represent dynamically adjustable weight coefficients.

[0117] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a fusion program for traditional Chinese medicine multi-modal examination information, and when the fusion program for traditional Chinese medicine multi-modal examination information is executed by a processor, the steps of the above-mentioned fusion method for traditional Chinese medicine multi-modal examination information are realized.

[0118] In summary, the present invention provides a method and related device for fusing traditional Chinese medicine multi-modal examination information. The method includes: obtaining multi-modal diagnosis and treatment data, decomposing each multi-modal diagnosis and treatment data into shared information and unique information within each modality; based on the disentanglement strategy, regarding the shared information between multiple modalities and the unique information of each modality as different categories, and fully learning the unique information of different modalities through a classifier in a new projection space; concatenating the learned shared information and unique information to obtain fused features, and through classification and multiple optimization iterations, enhancing the representation effect of the fused features on the disease. The present invention projects multi-modal diagnosis and treatment data into shared information and unique information through subspace mapping, designs a constraint function and a classification model through the disentanglement strategy to enhance the unique information and common information, serializes the common information and unique information, and through classification and multiple optimization iterations, further enhances the unique and common information of the information, achieving a good multi-modal feature representation effect.

[0119] It should be noted that in this article, the term "including", "comprising" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or terminal including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal. Without further limitation, an element defined by the statement "including a..." does not exclude the existence of another identical element in the process, method, article or terminal including the element.

[0120] Of course, those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware (such as a processor, a controller, etc.) through a computer program. The program can be stored in a computer-readable storage medium that can be read by a computer. When the program is executed, it can include the processes of the above method embodiments. The computer-readable storage medium can be a memory, a magnetic disk, an optical disc, etc.

[0121] It should be understood that the application of the present invention is not limited to the above examples. For those of ordinary skill in the art, improvements or transformations can be made according to the above description. All such improvements and transformations should fall within the protection scope of the appended claims of the present invention.

Claims

1. A method for fusing multi-modal diagnosis information in traditional Chinese medicine, characterized in that, The fusion method of the multi-modal diagnosis information in traditional Chinese medicine includes: Obtain multi-modal diagnosis and treatment data, and decompose each modal diagnosis and treatment data into common information and unique information within each modality; Based on the disentanglement strategy, regard the common information among multiple modalities and the unique information of each modality as different categories, and fully learn the unique information of different modalities through a classifier in a new projection space; Concatenate the learned common information and unique information to obtain a fusion feature, and enhance the representation effect of the fusion feature on the disease through classification and multiple optimization iterations; The obtaining of multi-modal diagnosis and treatment data and the decomposition of each modal diagnosis and treatment data into common information and unique information within each modality specifically include: If there are v modalities, then: ； Among them, X v represents the v-th modality of a batch of samples; d v is the dimension of the v-th modality; n is the number of samples used for training; V represents the number of modalities; represents the vector space; Decompose the diagnosis and treatment data of each modality into common information and unique information ; Common information: , representing the part that the v-th modality shares and is similar to other modalities; Exclusive information: , representing the part that the v-th modality has but other modalities do not have; where q represents a dimension after the sparse dictionary; After obtaining multi-modal diagnosis and treatment data and decomposing each modal diagnosis and treatment data into common information and unique information within each modality, it further includes: Project X v The projection is as follows: Formula (1); Among them, M v represents a sparse dictionary; is the F-norm; Said projecting X v After that, it further includes: Use indicate , the projection function of the modal information is optimized as follows: Formula (2); Among them, represents the projection function; represents the sparse matrix of the unique information after the decomposition of the v-th modality; represents the sparse matrix of the common information after the decomposition of the v-th modality; If the common information among multiple modalities is Z, the common information of the multi-modalities is expressed as: Formula (3); The step of, based on the disentanglement strategy, regarding the common information among multiple modalities and the unique information of each modality as different categories and fully learning the unique information of different modalities through a classifier in a new projection space specifically includes: Regarding the common information Z among multiple modalities and the unique information of each modality as different categories, the label of the category is , and in the new projection space, the classifier P is used to fully learn the unique information of different modalities. The learning strategy is as follows: Formula (4); The step of concatenating the learned common information and unique information to obtain a fusion feature and enhancing the representation effect of the fusion feature on the disease through classification and multiple optimization iterations specifically includes: Concatenate the learned common information and unique information, and sequentially combine them into a column vector or a row vector. By designing a classifier U, establish a mapping function between the fusion feature and the disease category Y: Formula (5); Combining formula (1), formula (2), formula (3), formula (4) and formula (5), the final multi-modal medical diagnosis data fusion function is expressed as: Formula (6); Among them, , and represent dynamically adjustable weight coefficients.

2. A fusion system for multi-modal diagnosis information in traditional Chinese medicine, characterized in that The fusion system of the multi-modal diagnosis information in traditional Chinese medicine is applied to the fusion method of the multi-modal diagnosis information in traditional Chinese medicine as claimed in claim 1. The fusion system of the multi-modal diagnosis information in traditional Chinese medicine includes: A data decomposition module, configured to obtain multi-modal diagnosis and treatment data, and decompose each modal diagnosis and treatment data into common information and unique information within each modality; An information learning module, configured to, based on the disentanglement strategy, regard the common information among multiple modalities and the unique information of each modality as different categories, and fully learn the unique information of different modalities through a classifier in a new projection space; A feature fusion module, configured to concatenate the learned common information and unique information to obtain a fusion feature, and enhance the representation effect of the fusion feature on the disease through classification and multiple optimization iterations.

3. A terminal, characterized in that, The terminal includes: a memory, a processor, and a fusion program of the multi-modal diagnosis information in traditional Chinese medicine stored on the memory and executable on the processor. When the fusion program of the multi-modal diagnosis information in traditional Chinese medicine is executed by the processor, the steps of the fusion method of the multi-modal diagnosis information in traditional Chinese medicine as claimed in claim 1 are implemented.

4. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a fusion program of the multi-modal diagnosis information in traditional Chinese medicine. When the fusion program of the multi-modal diagnosis information in traditional Chinese medicine is executed by a processor, the steps of the fusion method of the multi-modal diagnosis information in traditional Chinese medicine as claimed in claim 1 are implemented.

Images