Essence recommendation method based on multi-modal adaptive fusion, medium and equipment

Abstract

The invention discloses a syndrome element recommendation method based on multi-modal adaptive fusion, a medium and equipment. The method comprises the following steps: respectively extracting independent features of a tongue coating / face diagnosis image, a pulse diagnosis signal and a symptom text through pre-training CNN, LSTM and BERT networks; carrying out pixel-by-pixel sparse processing by adopting learnable parameters so as to eliminate redundancy and retain key features; dynamically fusing multi-modal features based on an attention mechanism, and adaptively adjusting the contribution degree of each modal; and finally, evidence element discrimination is realized through a full-connection classifier. The modal specificity is kept through independent feature extraction, the sparse processing and dynamic fusion mechanism are combined, the problems of feature interference and information loss in a traditional method are effectively solved, the accuracy and clinical adaptability of syndrome element recommendation are remarkably improved, and the stable performance can still be kept under the condition that part of modals are missing.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence, in particular to a syndrome element recommendation method based on multi-modal adaptive fusion, a medium and equipment. BACKGROUND

[0002] In the current intelligent diagnosis system of traditional Chinese medicine, effective fusion of multi-modal medical data is the key to improving the accuracy of syndrome element recommendation. The existing technology mainly has the following two typical fusion schemes and their inherent defects:

[0003] The first is a feature-level splicing fusion scheme, which specifically is to use independent feature extraction networks to process different modal data respectively, and to simply splice the extracted feature vectors. This method has the following problems: (1) feature redundancy interference: the physical meaning and numerical distribution difference of different modal features cause a large number of invalid interactions in the feature space after splicing, affecting the capture of key discriminative features by the model; (2) modal characteristic conflict: the dimension aliasing of image local correlation features and text global semantic features when splicing reduces the explainability of the features.

[0004] The second is a shared feature space fusion scheme, which specifically is to map multi-modal data to the same feature space through a unified encoder. This method has the following disadvantages: (1) loss of specific information: forced feature alignment can weaken the exclusive characteristics of each modality, for example, the time sequence dynamic features of pulse diagnosis signals are excessively smoothed in the unified encoding process; (2) poor clinical adaptability: fixed architecture is difficult to cope with the situation of missing part of the modal data in actual diagnosis and treatment (such as significant performance decline of the system when only tongue image is collected).

[0005] The existing methods have a fundamental contradiction between "maintaining modality specificity" and "implementing effective feature interaction": feature splicing retains modality characteristics but introduces noise interference, and shared encoding suppresses noise but loses modality exclusive information. This contradiction makes it difficult for existing systems to meet the needs of multi-modal data collaborative analysis in traditional Chinese medicine syndrome differentiation. SUMMARY

[0006] In view of the above problems, the present application provides a syndrome element recommendation method based on multi-modal adaptive fusion to solve the technical problems of feature redundancy interference, modal characteristic conflict, loss of specific information, poor clinical adaptability and the like existing in the existing syndrome element recommendation method.

[0007] To achieve the above purpose, in a first aspect, the present application provides a syndrome element recommendation method based on multi-modal adaptive fusion, which comprises:

[0008] S1: using a pre-trained convolutional neural network to extract first latent features from image data, the image data including tongue coating images and / or facial diagnosis images;

[0009] S2: using a long short-term memory network LSTM to extract second potential features of pulse diagnosis data;

[0010] S3: using a pre-trained BERT network to extract third potential features of symptom text data;

[0011] S4: performing pixel-by-pixel sparse processing on the first potential features, the second potential features and the third potential features respectively through learnable parameters to extract fine-grained information corresponding to different modal potential features;

[0012] S5: performing dynamic feature fusion on the fine-grained information corresponding to the different modal potential features to obtain global fusion features, and the dynamic feature fusion is obtained by using weighted summation based on attention weights;

[0013] S6: discriminating the global fusion features through a fully connected classifier to obtain representation information with syndrome classification discriminability.

[0014] Further, the pre-trained convolutional neural network is a ResNet50 network, and the first potential features of the image data are extracted using the pre-trained convolutional neural network, which is calculated by formula (1) as follows:

[0015] z image =ResNet(x image );

[0016] wherein x image represents image data, and z image represents the extracted first potential features.

[0017] Further, the pulse diagnosis data includes pulse diagnosis signal time series data, and the step S2 is calculated by formula as follows:

[0018] z pulse =LSTM(x pulse );

[0019] wherein x pulse represents pulse diagnosis signal time series data, z pulse represents the second potential features, and LSTM represents a long short-term memory network function.

[0020] Further, the step S3 is calculated by formula as follows:

[0021] z text =Bert(x text );

[0022] wherein x text represents symptom text data, and z text represents the third potential features, and Bert represents a BERT network function.

[0023] Further, step S4 is calculated as follows:

[0024] z i =z image ·W1;

[0025] z p =z pulse ·W2;

[0026] z t =z text ·W3;

[0027] wherein z image represents the first latent feature, W1 represents a learnable parameter of the same dimension as z image , z pulse represents the second latent feature, W2 represents a learnable parameter of the same dimension as z pulse , z text represents the third latent feature, W3 represents a learnable parameter of the same dimension as z text , the symbol “·” represents element-wise multiplication of vectors on the left and right, and z i represents the fine-grained information corresponding to the first latent feature, z p represents the fine-grained information corresponding to the second latent feature, and z t represents the fine-grained information corresponding to the third latent feature.

[0028] Further, step S5 includes the following steps:

[0029] performing dynamic feature fusion on the fine-grained information corresponding to the different modal latent features to obtain a fused latent feature, calculated as follows: z = CONCAT(α1·z image ,α2·z pulse ,α3·z text ), wherein α1, α2, and α3 represent learnable parameters for estimating the information amount of the modal, z represents the fused latent feature, and CONCAT(·) represents a concatenation function;

[0030] optimizing the fused latent feature through a fully connected neural network and mapping the optimized fused latent feature to a feature space of the original data to obtain the global fused feature, calculated as follows:

[0031] z final =FC(z);

[0032] wherein FC represents a fully connected neural network function, and z final represents the global fused feature.

[0033] Further, the representation information with the syndrome classification discriminability is calculated by a classifier, and the calculation formula is as follows:

[0034] L final = -ylogf(z final );

[0035] Wherein, L final represents the comprehensive error, the smaller the error, the more accurate the current syndrome class classification, f(·) represents the classifier function, and y represents the real syndrome class information.

[0036] Further, the method further comprises:

[0037] The comprehensive error is constrained according to the constraint condition, and the calculation formula is as follows:

[0038]

[0039] Wherein, L is the constrained comprehensive error, M is the number of modes, m represents the number of modes, ||·|| represents a norm, and W m represents the learnable parameters corresponding to the mth mode.

[0040] In a second aspect, the present application provides a computer readable storage medium having a computer program stored thereon, wherein the program is executed by a processor to implement the syndrome recommendation method based on multi-modal adaptive fusion as described in the first aspect of the present application.

[0041] In a third aspect, the present application provides an electronic device having a computer program stored thereon, comprising a processor and a storage medium, wherein the storage medium has a computer program stored thereon, and the computer program is executed by the processor to implement the syndrome recommendation method based on multi-modal adaptive fusion as described in the first aspect of the present application.

[0042] Unlike the prior art, the above technical solution relates to a syndrome recommendation method based on multi-modal adaptive fusion, medium and device, which comprises the following steps: independent features of tongue fur / facial diagnosis image, pulse diagnosis signal and symptom text are extracted by pre-training CNN, LSTM and BERT network respectively; learnable parameters are used for pixel-by-pixel sparse processing to eliminate redundancy and retain key features; multi-modal features are dynamically fused based on attention mechanism, and the contribution of each mode is adaptively adjusted; and finally, a full connection classifier is used to realize syndrome discrimination. The present application maintains modal specificity through independent feature extraction, effectively solves the problems of feature interference and information loss in traditional methods by combining sparse processing and dynamic fusion mechanism, significantly improves the accuracy and clinical adaptability of syndrome recommendation, and still maintains stable performance in the case of missing part of the mode.

[0043] The above summary related to the invention is only a summary of the technical solutions of the present application. In order to enable those skilled in the art to more clearly understand the technical solutions of the present application, further implement the contents recorded in the specification and drawings, and enable the above and other purposes, characteristics and advantages of the present application to be more easily understood, the following will be described in combination with the specific embodiments of the present application and the drawings. BRIEF DESCRIPTION OF DRAWINGS

[0044] The drawings are only used to show the principles, implementation manners, applications, characteristics and effects of the specific embodiments of the present application and other related contents, and cannot be considered as limitations of the present application.

[0045] In the drawings of the specification:

[0046] Figure 1 The flow chart of the evidence element recommendation method based on multi-modal adaptive fusion described for the first exemplary embodiment of the present application;

[0047] Figure 2 The flow chart of the evidence element recommendation method based on multi-modal adaptive fusion described for the second exemplary embodiment of the present application;

[0048] Figure 3 The flow chart of the evidence element recommendation method based on multi-modal adaptive fusion described for the third exemplary embodiment of the present application;

[0049] Figure 4 The module schematic diagram of the evidence element recommendation system related to an exemplary embodiment of the present application;

[0050] Figure 5 The schematic diagram of the electronic device described for an exemplary embodiment of the present application;

[0051] The reference signs related to the above drawings are explained as follows:

[0052] 10, electronic device;

[0053] 101, processor;

[0054] 102, storage medium;

[0055] 20, evidence element recommendation system;

[0056] 201, image feature extraction module;

[0057] 202, pulse diagnosis feature extraction module;

[0058] 203, text feature extraction module;

[0059] 204, sparse processing module;

[0060] 205, dynamic fusion module;

[0061] 206, classification determining module. DETAILED DESCRIPTION

[0062] To make the possible application scenarios, technical principles, specific schemes that can be implemented, purposes and effects achieved, etc. of the present application clear, the following will be described in detail in combination with the specific embodiments listed and the accompanying drawings. The embodiments described herein are only used to more clearly illustrate the technical schemes of the present application, and therefore only serve as examples, but cannot limit the protection scope of the present application.

[0063] In this article, the term "embodiment" means that the specific features, structures or characteristics described in combination with the embodiment can be included in at least one embodiment of the present application. The term "embodiment" appearing at various positions in the specification does not necessarily refer to the same embodiment, and does not particularly limit the independence or association between other embodiments. In principle, in the present application, as long as there is no technical contradiction or conflict, the technical features mentioned in each embodiment can be combined in any way to form a corresponding implementable technical scheme.

[0064] Unless otherwise defined, the meanings of the technical terms used herein are the same as those commonly understood by those skilled in the art to which the present application belongs; the use of related terms in this article is only for the purpose of describing specific embodiments, and is not intended to limit the present application.

[0065] In the description of the present application, the phrase "and / or" is a description of the logical relationship between the objects, which means that there can be three relationships, for example, A and / or B, which means that there are three cases: A exists, B exists, and A and B exist at the same time. In addition, the character " / " in this article generally represents that the associated objects before and after are a "or" logical relationship.

[0066] In the present application, the phrases such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual quantity, primary and secondary or order relationship between the entities or operations.

[0067] Without more limitations, in the present application, the phrases "include", "contain", "have" or other similar open expressions used in the statement are intended to cover non-exclusive inclusion, and these expressions do not exclude the presence of other elements in the process, method or product including the described elements, so that the process, method or product including a series of elements can not only include those limited elements, but also include other elements not explicitly listed, or also include the elements inherent to such process, method or product.

[0068] The same as the understanding in the "Examination Guidelines", in the present application, "greater than", "less than", "exceeding" and the like are understood as not including the number; "above", "below", "within" and the like are understood as including the number. In addition, in the description of the embodiments of the present application, the meaning of "multiple" is two or more (including two), and similar expressions related to "multiple" are also understood in this way, for example, "multiple groups", "multiple times" and the like, unless otherwise explicitly and specifically limited.

[0069] In the description of the embodiments of the present application, the spatially related expressions used, such as "center", "longitudinal", "transverse", "length", "width", "thickness", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "perpendicular", "top", "bottom", "inner", "outer", "clockwise", "counterclockwise", "axial", "radial", "circumferential", and the like, indicate the orientation or positional relationship based on the orientation or positional relationship shown in the specific embodiments or the drawings, and are only for the convenience of describing the specific embodiments of the present application or for the reader to understand, and do not indicate or imply that the indicated device or component must have a specific position, a specific orientation, or be constructed or operated in a specific orientation, and therefore cannot be understood as a limitation on the embodiments of the present application.

[0070] Unless otherwise explicitly specified or limited, in the description of the embodiments of the present application, the terms "mount", "connect", "connect", "fix", "set" and the like should be understood broadly. For example, the "connection" can be fixed connection, or detachable connection, or integrated setting; it can be mechanical connection, or electrical connection, or communication connection; it can be direct connection, or indirect connection through intermediate medium; it can be the communication or interaction relationship between two elements. For those skilled in the art to which the present application belongs, the specific meaning of the above terms in the embodiments of the present application can be understood according to the specific circumstances.

[0071] In the first aspect, as shown in the present application provides a kind of based on multi-modal adaptive fusion evidence recommendation method, the method comprises: Figure 1

[0072] S1: using pre-trained convolutional neural network to extract first latent feature from image data.

[0073] In step S1, the image data includes tongue fur image and / or facial diagnosis image, after acquiring image data, the image can be normalized and size adjusted first, so that it is converted into a standard format, facilitating subsequent feature extraction.

[0074] The tongue fur image extracts local texture features such as tongue color, tongue fur thickness and cracks, and the facial diagnosis image is directed to global features such as facial color and texture. The first latent feature is a high-dimensional tensor, which is reduced in dimension by global average pooling and then processed.​

[0075] S2: extracting second latent features of the pulse diagnosis data using a long short-term memory network (LSTM).

[0076] In step S2, the pulse diagnosis signal is time series data, which can be preprocessed, including denoising (which can be completed using a wavelet transform algorithm), segmentation (sliding window), and normalization.

[0077] Then, a bidirectional LSTM is used to capture the temporal dependence, and the input layer data of the LSTM includes the amplitude, frequency, and other features of the pulse wave at each time step, and the output layer data includes the second latent features, which are the hidden state at the last time step or the average pooling result of all time steps.

[0078] S3: extracting third latent features of the symptom text data using a pre-trained BERT network.

[0079] In step S3, the symptom text data contains traditional Chinese medicine symptom text information, such as “fear of cold, weakness, pale tongue”, which needs to be segmented into word vectors, and then a Chinese pre-trained BERT model is used to extract the hidden state of the [CLS] marker or the mean of all word vectors as the third latent features. Preferably, key symptom words can also be annotated by combining entity recognition, and the corresponding word vectors are weighted. For example, the input symptom text data is “fear of cold, pulse is fine and slow”, which can output a vector of a predetermined dimension after processing by the BERT network, and the word vector corresponding to “pulse is fine and slow” is given a higher attention weight.

[0080] S4: performing pixel-by-pixel sparse processing on the first, second, and third latent features using learnable parameters to extract fine-grained information corresponding to different modal latent features.

[0081] S5: dynamically fusing the fine-grained information corresponding to the different modal latent features to obtain global fusion features.

[0082] In step S5, the dynamic feature fusion is obtained by weighted summation based on attention weights, and the weights corresponding to each modal data can be calculated by a multi-layer perceptron (MLP), which is a feedforward neural network composed of at least three layers (input layer, hidden layer, and output layer), each layer containing multiple neurons, and achieving complex mapping through a nonlinear activation function. When some modal data is missing (e.g., no pulse diagnosis data), its weight α i automatically approaches 0, and only relies on the remaining modalities during dynamic feature fusion. For example, the attention weights of tongue diagnosis, pulse diagnosis, and text features are 0.4, 0.3, and 0.3, respectively, and when pulse diagnosis is missing, the weights are adjusted to 0.5, 0, and 0.5.

[0083] S6: The global fusion features are discriminated by a fully connected classifier to obtain representation information with syndrome classification discriminability.

[0084] In step S6, the fully connected classifier includes 2-3 layers of fully connected layers, the intermediate layer uses ReLU activation and Dropout regularization, and the output layer uses Softmax. The output layer nodes correspond to the classification results of TCM syndromes such as "qi deficiency" and "blood stasis", supporting multi-label classification.

[0085] The independent features of tongue fur / facial diagnosis images, pulse diagnosis signals and symptom texts are extracted by pre-training CNN, LSTM and BERT networks respectively; a learnable parameter is used for pixel-by-pixel sparse processing to eliminate redundancy and retain key features; the multi-modal features are dynamically fused based on an attention mechanism to adaptively adjust the contribution of each modality; and finally, a fully connected classifier is used to realize syndrome discrimination. The present application maintains modality specificity through independent feature extraction, effectively solves the problems of feature interference and information loss in traditional methods by combining sparse processing and dynamic fusion mechanism, significantly improves the accuracy and clinical adaptability of syndrome recommendation, and still maintains stable performance in the case of missing part of the modalities.

[0086] In some embodiments, the pre-trained convolutional neural network is a ResNet50 network, and the first latent feature is extracted from the image data using the pre-trained convolutional neural network by formula (1), which is expressed as follows:

[0087] z image =ResNet(x image );

[0088] wherein x image represents image data, and z image represents the extracted first latent feature.

[0089] The pulse diagnosis data includes pulse diagnosis signal time series data, and the formula for step S2 is as follows:

[0090] z pulse =LSTM(x pulse );

[0091] wherein x pulse represents pulse diagnosis signal time series data, z pulse represents the second latent feature, and LSTM represents a long short-term memory network function.

[0092] In some embodiments, the formula for step S3 is as follows:

[0093] z text =Bert(x text );

[0094] wherein x text represents the symptom text data, z text represents the third latent feature, and Bert represents a BERT network function.

[0095] In some embodiments, step S4 is calculated according to the following formula:

[0096] z i = z image · W1;

[0097] z p = z pulse · W2;

[0098] z t = z text · W3;

[0099] wherein z image represents the first latent feature, W1 represents a learnable parameter of the same dimension as z image , z pulse represents the second latent feature, W2 represents a learnable parameter of the same dimension as z pulse , z text represents the third latent feature, W3 represents a learnable parameter of the same dimension as z text , the symbol “·” represents element-wise multiplication of vectors on the left and right, z i represents the fine-grained information corresponding to the first latent feature, z p represents the fine-grained information corresponding to the second latent feature, and z t represents the fine-grained information corresponding to the third latent feature.

[0100] In some embodiments, as shown in Figure 3 , step S5 includes the following steps:

[0101] S31: performing dynamic feature fusion on the fine-grained information corresponding to the different modal latent features to obtain a fused latent feature.

[0102] The calculation formula of step S31 is as follows: z = CONCAT (a1 · z image , a2 · z pulse , a3 · z text ), wherein a1, a2, and a3 represent learnable parameters for estimating the information amount of the modal, z represents the fused latent feature, and CONCAT(·) represents a concatenation function.

[0103] S32: The fusion latent features are optimized using a fully connected neural network, and the optimized fusion latent features are mapped to the feature space of the original data to obtain the global fusion features. Here, the original data refers to the initial input pulse diagnosis signal, image data, and symptom text data.

[0104] The calculation formula for step S32 is as follows:

[0105] z final =FC(z);

[0106] Where FC represents a fully connected neural network function, Z final This indicates global fusion features.

[0107] In some embodiments, the representation information with element classification discriminative properties is calculated by a classifier, and the calculation formula is as follows:

[0108] L final =-ylogf(z) final );

[0109] Among them, L final The error represents the overall error. The smaller the error, the more accurate the classification of the current evidence element. f(·) represents the classifier function, and y represents the true evidence element category information.

[0110] In some embodiments, the method further includes:

[0111] The comprehensive error is constrained by the constraints, and the calculation formula is as follows:

[0112]

[0113] Where L is the constrained comprehensive error, M is the number of modes, m represents the mode number, ||·|| represents the first norm, and W m This represents the learnable parameters corresponding to the m-th mode.

[0114] like Figure 2 As shown, the model involved in this application processes data through the following steps: extracting image information features (i.e., the first latent features), extracting signal information features (i.e., the second latent features), and extracting text information features (i.e., the third latent features); establishing a learnable initial feature vector; dynamically fusing multimodal features; then iteratively training the model; and judging whether the model has been optimized based on the comprehensive error. If not, iterative training continues until optimization is complete. Then, by inputting pulse diagnosis data, symptom text data, and tongue coating or facial image data, a syndrome element recommendation system is used to obtain personalized syndrome element recommendations suitable for the current patient's symptoms and signs.

[0115] In a second aspect, the present application also provides a computer readable storage medium, having stored thereon a computer program, which, when executed by a processor, implements the evidence recommendation method based on multi-modal adaptive fusion as described in the first aspect of the present application.

[0116] The computer readable storage medium can be a volatile memory or a non-volatile memory, and can also include both volatile and non-volatile memories.

[0117] The non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a ferromagnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD ROM).

[0118] The volatile memory may be random access memory (RAM), which serves as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synclink dynamic random access memory (SLDRAM), and direct memory bus random access memory (DRRAM). The computer-readable storage media described in the embodiments of this application are intended to include these and any other suitable types of memory.

[0119] like Figure 5 As shown, in a third aspect, this application provides an electronic device 10, including a processor 101 and a storage medium 102, wherein a computer program is stored on the storage medium, and the computer program, when executed by the processor, implements the evidence recommendation method based on multimodal adaptive fusion as described in the first aspect of this application.

[0120] In some embodiments, the processor can be implemented by software, hardware, firmware or a combination thereof, and can use at least one of circuit, single or multiple Application Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Digital Signal Processing Device (DSPD), Programmable Logic Device (PLD), Field Programmable Gate Array (FPGA), Central Processing Unit (CPU), controller, microcontroller, microprocessor, so that the processor can execute part or all of the steps or any combination of steps in the multi-modal adaptive fusion based syndrome element recommendation method in various embodiments of the present application.

[0121] As shown in Figure 4 In a fourth aspect, the present application also provides a multi-modal adaptive fusion based syndrome element recommendation system 20, which comprises:

[0122] An image feature extraction module 201 is configured to process the tongue coating image and / or the facial diagnosis image by a pre-trained convolutional neural network to extract first latent features;

[0123] A pulse diagnosis feature extraction module 202 is configured to process the pulse diagnosis data by a Long Short-Term Memory (LSTM) network to extract second latent features;

[0124] A text feature extraction module 203 is configured to process the symptom text data by a pre-trained BERT network to extract third latent features;

[0125] A sparsification processing module 204 is configured to perform pixel-wise sparsification processing on the first latent features, the second latent features and the third latent features respectively by learnable parameters to extract fine-grained information of different modalities;

[0126] A dynamic fusion module 205 is configured to perform weighted summation on the fine-grained information based on attention weights to generate global fusion features;

[0127] A classification and discrimination module 206 is configured to process the global fusion features by a fully connected classifier to output a syndrome element classification result.

[0128] In this embodiment, the image feature extraction module uses ResNet or VGG as a pre-trained convolutional neural network. The LSTM network of the pulse diagnosis feature extraction module includes a bidirectional LSTM layer for capturing the temporal dependence of the pulse diagnosis signal. The sparsification processing module implements feature selection through a differentiable sparsification function, retaining key discriminative information. The attention weights of the dynamic fusion module are calculated by a multi-layer perceptron (MLP) and dynamically adjusted according to the input data. The classification discrimination module uses a Softmax activation function to output the syndrome probability distribution.

[0129] The present application proposes a syndrome recommendation system scheme based on multi-modal adaptive fusion, aiming to learn to effectively predict symptom information. Unlike existing syndrome recommendation system methods, the scheme proposed in the present application can learn the preference patterns of symptoms from multi-modal data. Specifically, the present application uses multiple feature extraction networks to extract features from multi-modal data, enabling it to provide personalized information for different features, and thus produce more distinctive patterns. This optimization enables the recommendation system to better identify the degree of association of the recommendations.

[0130] Finally, it should be noted that although the above embodiments have been described in the specification and drawings of the present application, they do not limit the scope of patent protection of the present application. Any technical solutions obtained by replacing or modifying the equivalent structures or equivalent processes based on the essential concept of the present application, using the contents described in the specification and drawings of the present application, and directly or indirectly implementing the technical solutions of the above embodiments in other related technical fields, etc., are all included within the scope of patent protection of the present application.

Claims

1. A method for recommending evidence based on multimodal adaptive fusion, characterized in that, The method includes: S1: Extract first latent features from image data, including tongue images and / or facial diagnosis images, using a pre-trained convolutional neural network; S2: Use a Long Short-Term Memory (LSTM) network to extract the second latent feature of pulse diagnosis data; S3: Use a pre-trained BERT network to extract the third latent feature from symptom text data; S4: The first latent feature, the second latent feature and the third latent feature are subjected to pixel-by-pixel sparsification using learnable parameters to extract fine-grained information corresponding to different modal latent features; S5: Dynamic feature fusion is performed on the fine-grained information corresponding to the different modal potential features to obtain global fused features. The dynamic feature fusion is obtained by weighted summation based on attention weights. S6: The global fusion features are discriminated using a fully connected classifier to obtain representation information with element classification discriminative properties.

2. The evidence recommendation method based on multimodal adaptive fusion as described in claim 1, characterized in that, The pre-trained convolutional neural network is a ResNet50 network. The first latent feature is extracted from the image data using the pre-trained convolutional neural network and calculated using formula (1). Formula (1) is expressed as follows: z image =ResNet(x image ); Where, x image Represents image data, z image This represents the first latent feature extracted.

3. The evidence recommendation method based on multimodal adaptive fusion as described in claim 1, characterized in that, The pulse diagnosis data includes pulse diagnosis signal time series data, and the calculation formula in step S2 is as follows: z pulse =LSTM(x pulse ); Where, x pulse This represents the time series data of pulse diagnosis signals, z pulse The second latent feature is represented by LSTM, which stands for Long Short-Term Memory Network Function.

4. The evidence recommendation method based on multimodal adaptive fusion as described in claim 1, characterized in that, The calculation formula for step S3 is as follows: z text =Bert(x text ); Where, x text Represents symptom text data, z text The third latent feature is represented by Bert, which represents the BERT network function.

5. The evidence recommendation method based on multimodal adaptive fusion as described in claim 1, characterized in that, The calculation formula for step S4 is as follows: With i =z image ·W1; With p =z pulse ·W2; z t =z text ·W3; Among them, z image W1 represents the first latent feature, and W1 represents the relationship with z. image Learnable parameters of the same dimension, z pulse W2 represents the second latent feature, and W2 represents the relationship with z. pulse Learnable parameters of the same dimension, z text W3 represents the third latent feature, and W3 represents the relationship with z. text Learnable parameters of the same dimension, the symbol "·" indicates element-wise multiplication of the vectors on both sides, z i z represents the fine-grained information corresponding to the first latent feature. p z represents the fine-grained information corresponding to the second latent feature. t This represents the fine-grained information corresponding to the third latent feature.

6. The evidence element recommendation method based on multimodal adaptive fusion as described in claim 5, characterized in that, Step S5 includes the following steps: Dynamic feature fusion is performed on the fine-grained information corresponding to the different modal latent features to obtain the fused latent features. The calculation formula is as follows: z = CONCAT(α1·z) i ,α2·z p ,α3·z t ), where α1, α2, and α3 represent learnable parameters used to estimate the information content of the modality, z represents the fused latent features, and CONCAT(·) represents the concatenation function; The fusion latent features are optimized using a fully connected neural network, and the optimized fusion latent features are mapped to the feature space of the original data to obtain the global fusion features. The calculation formula is as follows: from final =FC(z); Where FC represents a fully connected neural network function, Z final This indicates global fusion features.

7. The evidence recommendation method based on multimodal adaptive fusion as described in claim 6, characterized in that, The discriminative information for element classification is calculated using a classifier, and the calculation formula is as follows: L final =-ylogf(z final ); Among them, L final The error represents the overall error. The smaller the error, the more accurate the classification of the current evidence element. f(·) represents the classifier function, and y represents the true evidence element category information.

8. The evidence element recommendation method based on multimodal adaptive fusion as described in claim 7, characterized in that, The method further includes: The comprehensive error is constrained by the constraints, and the calculation formula is as follows: Where L is the constrained comprehensive error, M is the number of modes, m represents the mode number, ||·|| represents the first norm, and W m This represents the learnable parameters corresponding to the m-th mode.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the evidence recommendation method based on multimodal adaptive fusion as described in any one of claims 1 to 8.

10. An electronic device having a computer program stored thereon, characterized in that, It includes a processor and a storage medium, wherein a computer program is stored on the storage medium, and the computer program, when executed by the processor, implements the evidence recommendation method based on multimodal adaptive fusion as described in any one of claims 1 to 8.

Images