Intelligent analysis method and system for fusing multi-modal clinical data

Abstract

The application discloses an intelligent analysis method and system for fusing multi-modal clinical data, and relates to the technical field of medical health information processing.The method comprises the following steps: obtaining the amplitude sequence, hand characteristic and pulse condition label of a diagnosed patient under different applied pressure levels, and grouping the amplitude sequence; obtaining the attention weight of each frequency band in each group, and the discrimination weight of each element in each frequency band in each group; weighting the first feature vector and the second feature vector based on the attention weight and the discrimination weight to generate the comprehensive feature vector of each amplitude sequence; constructing a pulse condition recognition model, constructing a training sample set based on the comprehensive feature vector, constructing a discrimination weight penalty mechanism, and iteratively optimizing the model; collecting the pulse diagnosis data of a patient to be diagnosed, and outputting the pulse condition label based on the pulse condition recognition model.The application can adaptively modulate the effect of different pressure levels on pulse condition information, and improve the accuracy of pulse condition prediction.

Description

Technical Field

[0001] This invention relates to the field of healthcare information processing technology, and in particular to an intelligent analysis method and system that integrates multimodal clinical data. Background Technology

[0002] The intelligent pulse diagnosis instrument is a smart device that simulates the pulse diagnosis process in Traditional Chinese Medicine (TCM) through sensor technology, big data analysis, and machine learning algorithms. It collects pulse signals from the cun, guan, and chi positions at the wrist and analyzes pulse characteristics (such as superficiality / deepness, slowness / rapidity, slipperiness / roughness, etc.) using TCM theory to help determine constitution, disease tendencies, or health status, thus improving the efficiency and standardization of pulse diagnosis. In TCM theory of pulse recognition, the eight elements of pulse (pulse location, pulse rate, pulse strength, pulse length, pulse width, pulse rhythm, pulse tension, and pulse flow) and twenty-eight pulse patterns are key features. The relationship between the eight elements and the twenty-eight pulse patterns is one of "underlying attributes" and "comprehensive form." The eight elements are the basic physical parameters constituting all pulse patterns, while the twenty-eight pulse patterns are specific diagnostic conclusions formed by combining these parameters in different dimensions. In practical applications, the intelligent pulse diagnosis instrument is usually used in conjunction with traditional manual pulse diagnosis methods. Based on the collected pulse signals, the intelligent pulse diagnosis instrument analyzes and outputs pulse pattern labels that display the patient's symptoms. Pulse labels typically consist of multiple pulse types from the twenty-eight types under the eight elements of pulse diagnosis. They are used to assist professional physicians in making judgments, and professional physicians examine and interpret the symptoms corresponding to the pulse information in the pulse labels.

[0003] The clinical data collected by intelligent pulse diagnosis instruments is essentially typical multimodal data. It includes not only pulse amplitude sequences collected at different pressure levels (superficial, medium, and deep), but also information on the corresponding hand (left or right) and collection site. The biological characteristics of the pulse are complex; the distribution and manifestation of pulse information contained in its eight elements vary across different frequency ranges. Applying different levels of pressure significantly alters the presentation of this pulse information; some elements are enhanced, while others are suppressed or masked. This dynamic information modulation effect caused by pressure makes it difficult for single-modal analysis or simple feature extraction methods to comprehensively and accurately characterize the complex biological features of the pulse.

[0004] For clinical data collected by intelligent pulse diagnosis instruments, existing analysis methods typically assign fixed attention weights to the eight pulse characteristics and rely on the energy value differences of amplitude sequences between fixed frequency bands to predict pulse characteristics. However, existing analysis methods ignore the significant differences in the degree to which different applied pressure levels enhance or suppress the information of each pulse characteristic, and lack a pressure-adaptive analysis mechanism. In practical applications, the true information of some pulse characteristics under different pressures may be masked by environmental noise or smoothed during feature fusion, leading to decreased accuracy and increased prediction bias, resulting in a higher probability of pulse misdiagnosis. Summary of the Invention

[0005] To enhance the ability of intelligent pulse diagnosis instruments to capture the eight elements of pulse and improve the accuracy of pulse prediction, and to address the problem of misdiagnosis caused by differences in the suppression of element information under different applied pressure levels, this invention provides an intelligent analysis method and system that integrates multimodal clinical data. The technical solution is as follows: In a first aspect, the present invention provides an intelligent analysis method for integrating multimodal clinical data, comprising the following steps: acquiring amplitude sequences, hand features, and pulse labels of diagnosed patients under different applied pressure levels, and grouping the amplitude sequences based on hand features and applied pressure levels; adaptively dividing the spectrum of the amplitude sequences within each group into frequency bands, obtaining the optimal frequency band segmentation scheme for each group, and obtaining the attention weight of each frequency band in each group based on the optimal frequency band segmentation result; performing secondary grouping of the amplitude sequences within each group based on the eight elements of pulse and pulse type, and calculating the differences between subgroups under each element within the same group in each frequency band range, so as to obtain the differences of each element in each group in each frequency band. The discrimination weights are calculated as follows: The amplitude of each frequency in the spectrum of the amplitude sequence is weighted based on the attention weight to obtain the first feature vector of the amplitude sequence; the amplitude of each frequency in the spectrum of the amplitude sequence is weighted based on the discrimination weight to obtain the second feature vector of the amplitude sequence relative to each element in its group; the first and second feature vectors are combined to generate a comprehensive feature vector for each amplitude sequence; a pulse recognition model is constructed, and a training sample set is built based on the comprehensive feature vector; the error between the model's prediction results and the actual pulse labels is calculated during training to construct a discrimination weight penalty mechanism for iterative optimization of the model; pulse diagnosis data of patients to be diagnosed are collected, and pulse labels are output based on the pulse recognition model.

[0006] Preferably, pulse signals from corresponding points on the patient's left and right hands are collected using an intelligent pulse diagnostic instrument at the same sampling frequency, with one sample collected at each of the three applied pressure levels: superficial, medium, and deep. The collected pulse signals are standardized to obtain six equal-length amplitude sequences. Historical data from multiple diagnosed patients are obtained from the historical records of the intelligent pulse diagnostic instrument. The historical data of each diagnosed patient includes three amplitude sequences corresponding to the left and right hands at the three applied pressure levels, and a pulse label. All amplitude sequences are grouped according to hand characteristics and applied pressure level to obtain six groups: left hand superficial, left hand medium, left hand deep, right hand superficial, right hand medium, and right hand deep.

[0007] Preferably, any group is selected as the target group, and a Fourier transform is performed on the amplitude sequences within the target group to obtain the spectrum corresponding to each amplitude sequence within the target group; a candidate set of frequency band segmentation schemes is preset, and any frequency band segmentation scheme within the set is selected as the target segmentation scheme; based on the target segmentation scheme, the frequency range of each spectrum is divided into three frequency bands: low, medium, and high; the standard deviation of the amplitude of each spectrum within the target group within the same frequency band is calculated, and the mean of the standard deviation is used as the information richness of the target group in that frequency band; similarly, the information richness of the target group in each frequency band is obtained; the cumulative value of the information richness of the target group in each frequency band is used as the evaluation score of the target segmentation scheme relative to the target group.

[0008] Preferably, the candidate set of frequency band segmentation schemes is traversed to obtain the evaluation score of each frequency band segmentation scheme relative to the target group. The frequency band segmentation scheme with the highest evaluation score is selected as the optimal frequency band segmentation scheme for the target group. Similarly, the optimal frequency band segmentation scheme for each group is obtained. Based on the optimal frequency band segmentation scheme for the target group, the frequency range of each spectrogram within the target group is divided to obtain the optimal frequency band segmentation result for the target group. Based on the optimal frequency band segmentation result for the target group, the optimal information richness of the target group in each frequency band is obtained. The normalized optimal information richness of the target group in each frequency band is used as the attention weight of each frequency band in the target group. Similarly, the attention weight of each frequency band in each group is obtained.

[0009] Preferably, based on the eight elements of pulse diagnosis and twenty-eight types of pulse diagnosis, each group is divided into eight large groups corresponding to the eight elements of pulse diagnosis and twenty-eight subgroups corresponding to the twenty-eight types of pulse diagnosis. Each large group contains multiple subgroups. Each amplitude sequence within the same group corresponds to a pulse diagnosis label. Amplitude sequences with the same pulse diagnosis type in the same group are classified into the corresponding subgroups. Each group is traversed to obtain multiple subgroups under each element within each group. Any one of the eight elements of pulse diagnosis is selected as the target element. Any two subgroups under the target element within the same group are extracted. An amplitude sequence is randomly selected from each of the two subgroups, and the root mean square error of the amplitude between the spectra of the two amplitude sequences within the same frequency band is calculated.

[0010] Preferably, the two subgroups are traversed to obtain the root mean square error of the amplitude values ​​of all amplitude sequences within the two subgroups within the same frequency band, and the mean of the root mean square error is used as the difference between the two subgroups in that frequency band; the mean of the differences between the subgroups under the target element in the same group within the same frequency band is used as the distinguishing feature value of the target element in that group in that frequency band, and similarly, the distinguishing feature value of the target element in each frequency band is obtained; the normalized distinguishing feature value of the target element in each frequency band within the same group is used as the distinguishing weight of the target element in each frequency band within the corresponding group, and each element in each group is traversed to obtain the distinguishing weight of each element in each frequency band within each group.

[0011] Preferably, based on the attention weights of each frequency band in the group, the amplitude values ​​of each frequency within the corresponding frequency band range in the spectrum diagram of each amplitude sequence in the group are weighted to obtain the first feature vector of each amplitude sequence in the group; based on the discrimination weights of each element in each frequency band in the group, the amplitude values ​​of each frequency within the corresponding frequency band range in the spectrum diagram of each amplitude sequence in the group are weighted to obtain the second feature vector of the amplitude sequence relative to each element in its group; each amplitude sequence corresponds to one first feature vector and eight second feature vectors, and the first feature vector and eight second feature vectors of each amplitude sequence are concatenated in the same order to obtain the comprehensive feature vector of each amplitude sequence.

[0012] Preferably, a multilayer perceptron structure is used to construct the pulse recognition model, which includes an input layer, three hidden fully connected layers, and an output layer. Based on hand characteristics, the comprehensive feature vectors corresponding to the left and right hands under different applied pressure levels, along with their corresponding pulse labels, are used as training samples. Each diagnosed patient corresponds to two training samples, and this process is repeated for all diagnosed patients to obtain the training sample set. During model training, the processing result of the third hidden fully connected layer is the predicted probability of each of the twenty-eight pulse types. For any training sample, the actual probability of each pulse type is extracted from its pulse label. Calculate the absolute difference between the actual probability and the predicted probability of each pulse type, and use the mean of the absolute difference as the prediction error of the training sample. Iterate through the training sample set and use the mean of the prediction errors of all training samples as the penalty coefficient for the high-frequency band. Based on the penalty coefficient for the high-frequency band, weight the discrimination weight of each element in each group in the high-frequency band to update the discrimination weight of each element in each group in each frequency band. Iteratively update the penalty coefficient, discrimination weight, comprehensive feature vector and training sample set for the high-frequency band to iteratively optimize the pulse recognition model until the preset number of iterations is reached and the optimization training stops.

[0013] Preferably, pulse signals from corresponding parts of the left and right hands of the patient to be diagnosed are collected using an intelligent pulse diagnostic instrument to obtain six amplitude sequences corresponding to the patient, which are then classified into corresponding groups. Fourier transform is performed on the six amplitude sequences to obtain the spectrum of each sequence. Based on the attention weights of each frequency band in each group, the first feature vector corresponding to the six amplitude sequences is obtained. Based on the discrimination weights of each element in each group in each frequency band obtained from the last iteration update, the second feature vector corresponding to the six amplitude sequences is obtained, thereby generating a comprehensive feature vector corresponding to the six amplitude sequences. Based on hand-specific features, the three comprehensive feature vectors corresponding to the left and right hands of the patient to be diagnosed are sequentially input into the pulse recognition model, outputting pulse labels corresponding to the left and right hands of the patient to be diagnosed, respectively.

[0014] Secondly, the present invention provides an intelligent analysis system for integrating multimodal clinical data to implement the above-mentioned intelligent analysis method for integrating multimodal clinical data, comprising: a processor, a memory, a communication interface, a data acquisition module and a display module in an intelligent pulse diagnostic instrument, wherein the processor stores computer program instructions for implementing the above-mentioned intelligent analysis method for integrating multimodal clinical data, and the communication interface is electrically connected to the data acquisition module and the display module in the intelligent pulse diagnostic instrument.

[0015] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention replaces fixed-band analysis with a group-adaptive dynamic frequency band division mechanism, enabling the model to more precisely capture the specific changes in pulse information under different hand types and pressure levels. By introducing a comprehensive adaptive discrimination weighting of the eight pulse elements and pressure level, it replaces the coarse approach of assigning the same weight to all elements, significantly enhancing the ability to capture the differentiated information of each element, thereby improving the overall accuracy of pulse prediction and reducing the risk of misdiagnosis. Furthermore, through a high-frequency noise adaptive penalty mechanism based on model prediction error, it can intelligently suppress high-frequency irrelevant information when the signal quality is poor or there is noise interference, while retaining the core pulse features of mid- and low-frequency frequencies. This greatly improves the objectivity, stability, and robustness of the pulse recognition model, ensuring the reliability of pulse recognition results in actual clinical applications. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating the implementation of the intelligent analysis method for integrating multimodal clinical data according to an embodiment of the present invention.

[0017] Figure 2 This is a structural block diagram of an intelligent analysis system that integrates multimodal clinical data, as described in an embodiment of the present invention. Detailed Implementation

[0018] The technical features of the present invention will be further described in detail below with reference to the accompanying drawings so that those skilled in the art can understand them.

[0019] Intelligent analysis methods that integrate multimodal clinical data, with the following implementation process: Figure 1 As shown, the specific implementation steps are as follows: Step S1: Obtain amplitude sequences, hand features, and pulse labels of confirmed patients under different applied pressure levels, and group the amplitude sequences based on hand features and applied pressure levels.

[0020] Specifically, the pulse signals of the corresponding parts of the patient's left and right hands were collected by the intelligent pulse diagnosis instrument at the same sampling frequency. The pulse signals were collected once each under three pressure levels: superficial, medium and deep. The collected pulse signals were standardized to obtain six equal-length amplitude sequences. By default, pulse signals from the same area on both hands of all patients are collected. For example, the collection frequency is 1000Hz, and the collection time for all patients is the same and not less than 1 minute.

[0021] The historical data of multiple diagnosed patients were obtained from the history of the intelligent pulse diagnosis instrument. Each diagnosed patient's historical data contained three amplitude sequences corresponding to the left and right hands under three applied pressure levels, and a pulse label. The pulse label was one or more of the twenty-eight pulse types. Each confirmed patient corresponds to six amplitude sequences and two pulse labels: three amplitude sequences and one pulse label for the left hand under three applied pressure levels, and three amplitude sequences and one pulse label for the right hand. The pulse label for each confirmed patient is a pulse recognition result determined after correction by a professional physician. Moreover, the pulse labels of multiple confirmed patients extracted from the historical records of the intelligent pulse diagnosis instrument need to include all twenty-eight pulse types to enhance the training effect of the pulse recognition model in subsequent steps and improve the prediction accuracy of the pulse recognition model.

[0022] All amplitude sequences were grouped according to hand characteristics and applied pressure level, resulting in six groups: left-hand floating, left-hand middle-hand, left-hand sinking, right-hand floating, right-hand middle-hand, and right-hand sinking. Among them, the six amplitude sequences of each confirmed patient were classified into six groups. The number of amplitude sequences in each group was the same, and each amplitude sequence in the group corresponded to a pulse label.

[0023] Step S2: Perform adaptive frequency band division on the spectrum of the amplitude sequence within each group, obtain the optimal frequency band segmentation scheme for each group, and obtain the attention weight of each frequency band in each group based on the optimal frequency band segmentation result.

[0024] In pulse diagnosis analysis in the fusion of multimodal clinical data, the pulse information contained in the eight pulse elements varies at different frequencies. Applying different levels of pressure by the physician can either enhance or suppress the pulse information of some elements. Most existing analysis methods predict pulse characteristics by assigning attention weights based on the energy value differences of amplitude sequences across frequency bands. This relies on the fixed information contained in the eight pulse elements within different frequency bands, neglecting the information suppression and noise effects caused by varying levels of applied pressure. Consequently, the actual pulse information is either ignored or underestimated.

[0025] Therefore, this step analyzes the information complexity contained in different frequency bands of each group based on the standard deviation of the amplitude in the spectrum graph of each group after grouping, so as to perform adaptive frequency band division and attention weight acquisition for groups, thereby improving the ability to capture information of the eight elements of pulse under different applied pressure levels. Furthermore, based on the information complexity of different frequency bands, attention weights are assigned to each frequency band to enhance the influence weight of high-quality information frequency bands (low-frequency and mid-frequency bands) and reduce the influence weight of high-noise frequency bands (high-frequency bands).

[0026] Specifically, any group is selected as the target group, and the amplitude sequences within the target group are subjected to Fourier transform to obtain the spectrum diagram corresponding to each amplitude sequence within the target group; a candidate set of frequency band segmentation schemes is preset, and any frequency band segmentation scheme within the set is selected as the target segmentation scheme; The process of constructing the candidate set of frequency band segmentation schemes is as follows: First, based on human experience, the value ranges of low-frequency segmentation nodes and high-frequency segmentation nodes are set, and the moving step size of the values ​​of the two segmentation nodes is set to generate the candidate value sets of low-frequency segmentation nodes and high-frequency segmentation nodes. The two segmentation nodes can divide the frequency range of the spectrum into three frequency bands: low, medium and high. Then, the candidate value sets of the two segmentation nodes are traversed in turn to divide the frequency range of the spectrum, thus obtaining the candidate set of frequency band segmentation schemes.

[0027] Based on the target segmentation scheme, the frequency range of each spectrogram is divided into three frequency bands: low, medium, and high. The standard deviation of the amplitude of each spectrogram within the target group in the same frequency band is calculated, and the mean of the standard deviation is used as the information richness of the target group in that frequency band. Similarly, the information richness of the target group in each frequency band is obtained. Among them, the standard deviation of the amplitude within the same frequency band measures the complexity of the pulse information contained in the frequency band. The larger the standard deviation, the greater the complexity and the greater the difference in the distribution of pulse information. The richer the pulse information contained in the corresponding frequency band, that is, the greater the information richness.

[0028] The cumulative value of the information richness of the target group in each frequency band is used as the evaluation score of the target segmentation scheme relative to the target group; The higher the evaluation score of the target group, the greater the overall information richness of pulse information contained in the three frequency bands divided by the target segmentation scheme, and the clearer the capture of pulse information in each frequency band, thereby avoiding pulse recognition errors caused by smoothing of pulse information.

[0029] Traverse the candidate set of frequency band partitioning schemes to obtain the evaluation score of each frequency band partitioning scheme relative to the target group. Select the frequency band partitioning scheme with the highest evaluation score as the optimal frequency band partitioning scheme for the target group. Similarly, obtain the optimal frequency band partitioning scheme for each group. Since most of the noise in pulse information is distributed in the high-frequency range, if the evaluation scores are the same, the frequency segmentation scheme with the largest low-frequency or mid-frequency range and the smallest high-frequency range will be selected as the optimal frequency segmentation scheme for the corresponding group.

[0030] Furthermore, based on the optimal frequency band segmentation scheme of the target group, the frequency range of each spectrum map within the target group is divided to obtain the optimal frequency band segmentation result of the target group; based on the optimal frequency band segmentation result of the target group, the optimal information richness of the target group in each frequency band is obtained, and the normalized optimal information richness of the target group in each frequency band is used as the attention weight of each frequency band in the target group. Similarly, the attention weight of each frequency band in each group is obtained. The higher the information richness value, the richer the pulse information contained in the target group within the corresponding frequency band. Therefore, the more attention needs to be paid to the corresponding frequency band within the target group, the greater the attention weight should be given to the corresponding frequency band in order to capture more pulse information. The optimal information richness of each frequency band within the same group is normalized by summation and normalization. For example, the optimal information richness of the low-frequency band within the target group is used as the numerator, the sum of the optimal information richness of the three frequency bands within the target group is used as the denominator, and the ratio of the two is used as the normalized optimal information richness of the low-frequency band, which is the attention weight of the low-frequency band within the target group. Similarly, the attention weights of the mid-frequency and high-frequency bands within the target group are obtained.

[0031] Step S3: Based on the eight elements of pulse and pulse type, the amplitude sequence within each group is regrouped, and the differences between subgroups under each element within the same group in each frequency band are calculated to obtain the discrimination weight of each element in each group in each frequency band.

[0032] Because the characteristics of the eight elements of pulse diagnosis vary in different frequency bands—for example, the information differences of different pulse rates are mainly in the low-frequency band, different pulse tensions are in the mid-frequency band, different pulse lengths are in the low-frequency or mid-frequency band, and pulse flow is in the mid-frequency or high-frequency band—the degree of inhibition and enhancement of pulse characteristics in different frequency bands differs when different levels of pressure are applied, which can easily lead to the actual information of the eight elements of pulse diagnosis being masked and smoothed out.

[0033] Therefore, this step regroups the amplitude sequences within each group based on the eight elements of pulse diagnosis, enabling independence analysis of the elements. By analyzing the differences in the spectrograms between different subgroups, the ability of the three frequency bands to distinguish the elements under different applied pressure levels is quantified. The multi-scale discrimination weights of the elements and applied pressure are obtained to further improve the ability to capture pulse information.

[0034] Specifically, based on the eight elements of pulse diagnosis and twenty-eight types of pulse diagnosis, each group is divided into eight major groups corresponding to the eight elements of pulse diagnosis and twenty-eight subgroups corresponding to the twenty-eight types of pulse diagnosis. Each major group contains multiple subgroups. Each amplitude sequence within the same group corresponds to a pulse diagnosis label. Amplitude sequences with the same pulse diagnosis type in the same pulse diagnosis label within the same group are classified into the corresponding subgroups. By traversing each group, multiple subgroups under each element within each group are obtained. For amplitude sequences containing multiple pulse types in the pulse label, they are classified into multiple corresponding subgroups, meaning that the same amplitude sequence may be divided into multiple subgroups.

[0035] In addition, any one of the eight elements of pulse diagnosis is selected as the target element, and any two subgroups under the target element in the same group are extracted. An amplitude sequence is randomly selected from the two subgroups, and the root mean square error of the amplitude between the two amplitude sequences in the same frequency band is calculated. Traverse the two subgroups, obtain the root mean square error of the amplitude values ​​of all amplitude sequences in the two subgroups within the same frequency band, and use the mean of the root mean square error as the difference between the two subgroups in that frequency band. The mean value of the differences between subgroups of the target element within the same group within the same frequency band is used as the distinguishing characteristic value of the target element in that group in that frequency band. Similarly, the distinguishing characteristic values ​​of the target element in each frequency band within the group are obtained. The root mean square error (RMSE) measures the intuitive difference in frequency domain amplitude between different pulse patterns under the same element. The greater the difference, the greater the difference in frequency domain amplitude between different pulse patterns under the same element within the corresponding frequency band. This also indicates that the frequency band has a greater ability to distinguish between different pulse patterns under the same element. In other words, the information on different pulse patterns under the same element contained in the frequency band within the corresponding group is clearer and more specific. The pulse pattern information about the element extracted from the frequency band is more reliable and should be trusted.

[0036] The normalized discrimination capability feature value of the target element in each frequency band within the same group is used as the discrimination weight of the target element in each frequency band within the corresponding group. By traversing each element in each group, the discrimination weight of each element in each group within each frequency band is obtained. In this process, the summation and normalization method is also used to normalize the distinguishing ability feature values ​​of target elements in each frequency band within the same group. For example, the distinguishing ability feature value of target elements in the low frequency band within the target group is used as the numerator, the sum of the distinguishing ability feature values ​​of target elements in the three frequency bands within the target group is used as the denominator, and the ratio of the two is used as the normalized distinguishing ability feature value in the low frequency band, which is the distinguishing weight of target elements in the low frequency band within the target group.

[0037] Step S4: Weight the amplitude of each frequency in the spectrum of the amplitude sequence based on attention weight to obtain the first feature vector of the amplitude sequence. Weight the amplitude of each frequency in the spectrum of the amplitude sequence based on discrimination weight to obtain the second feature vector of the amplitude sequence relative to each element in its group. Combine the first feature vector and the second feature vector to generate the comprehensive feature vector of each amplitude sequence.

[0038] Specifically, based on the attention weights of each frequency band in the group, the amplitudes of each frequency within the corresponding frequency band range in the spectrum of each amplitude sequence in the group are weighted to obtain the first feature vector of each amplitude sequence in the group; The physical meaning of the first feature vector is to reflect the weighted energy information of each frequency component of the pulse signal under specific hand characteristics and applied pressure levels. It amplifies the signal components in the frequency bands with rich information and high complexity under the hand characteristics and pressure scenarios corresponding to the group, and weakens the signal components in the frequency bands with flat information and small changes. Thus, it adaptively highlights the frequency components that are generally more informative and valuable for analysis under the specific hand characteristics and applied pressure levels corresponding to the group.

[0039] Based on the discrimination weight of each element in each frequency band within the group, the amplitude of each frequency within the corresponding frequency band range in the spectrum of each amplitude sequence in the group is weighted to obtain the second feature vector of the amplitude sequence relative to each element in its group. The physical meaning of the second feature vector is that, for a single pulse element (such as pulse rate, pulse tension, etc.), the feature representation obtained by weighting the original spectrogram for element discrimination under specific hand characteristics and applied pressure level is: the second feature vector amplifies the frequency components most important for identifying a certain element and suppresses unimportant or interfering frequency components, thereby "customizing" the focus of attention for each element, ensuring that even when different applied pressure levels have different modulations on the frequency band, the key identification information of each element can be extracted and enhanced individually.

[0040] Each amplitude sequence corresponds to a first feature vector and eight second feature vectors. The first feature vector and eight second feature vectors of each amplitude sequence are concatenated in the same order to obtain the comprehensive feature vector of each amplitude sequence. The physical meaning of the comprehensive feature vector is a multi-dimensional unified representation that integrates group scene features and all element enhancement features. Specifically, by fusing pressure adaptive frequency band information and element discrimination information, it forms a unified representation that includes both global pressure environment information and local element discrimination information. This provides a multi-view, multi-level feature map for the subsequent training of the pulse recognition model, encompassing both macroscopic hand identification and pressure scene background, as well as microscopic close-ups of various elements. This allows the pulse recognition model to simultaneously consider pressure modulation effects and element specificity, enabling more comprehensive and robust pulse prediction and improving its accuracy.

[0041] Step S5: Construct a pulse recognition model and build a training sample set based on the comprehensive feature vector. Calculate the error between the model's prediction results and the real pulse labels during the training process to construct a discrimination weight penalty mechanism and iteratively optimize the model.

[0042] Due to patient movement and environmental noise during pulse recognition, some elements have excessively high discriminative weights in the high-frequency band, leading to overemphasis on noise information and significant prediction bias. Therefore, this step constructs a penalty coefficient for the high-frequency band based on the deviation between the predicted and actual pulse labels. This penalty coefficient adaptively adjusts the discriminative weights of the high-frequency band to suppress noise influence, avoid loss of detail, and achieve accurate pulse recognition.

[0043] Specifically, a pulse recognition model is constructed using a multilayer perceptron structure, which includes an input layer, three hidden fully connected layers, and an output layer. Mean squared error is selected as the loss function. Based on hand features, the comprehensive feature vectors and corresponding pulse labels of the left and right hands under different applied pressure levels are used as training samples. Each diagnosed patient corresponds to two training samples. The training sample set is obtained by traversing all diagnosed patients. In this model, based on the commonly used multilayer perceptron model structure, the first hidden fully connected layer selects ReLU as the activation function and has an output size of 256, the second hidden fully connected layer selects ReLU as the activation function and has an output size of 128, the third hidden fully connected layer selects ReLU as the activation function and has an output size of 28, and the output layer selects the Sigmoid function as the activation function and has an output size of 1.

[0044] The pulse recognition model takes a comprehensive feature vector from the training samples as input and outputs a pulse label as output. The pulse recognition model is trained based on the training sample set. Among them, the comprehensive feature vector serves as the unified input to the pulse recognition model, enabling the model to automatically learn how to most effectively weigh hand features, stress scene information, and detailed information of various elements through training, thereby making the final comprehensive pulse judgment. Therefore, training the pulse recognition model based on the comprehensive feature vector can avoid the difficulty of manually designing decision rules, encode prior knowledge such as stress adaptation and element attention into the comprehensive feature vector, and hand it over to the data-driven pulse recognition model for final fusion and decision.

[0045] During the training of the model, the processing result of the third hidden fully connected layer is the predicted probability of each of the twenty-eight pulse types. For any pulse label, if a certain pulse type exists, the actual probability of that pulse type in the pulse label is marked as 1; otherwise, if a certain pulse type does not exist, the actual probability of that pulse type in the pulse label is marked as 0. For any training sample, extract the actual probability of each pulse type from its pulse label, calculate the absolute difference between the actual probability and the predicted probability of each pulse type, and use the mean of the absolute difference as the prediction error of the training sample. Iterate through the training sample set and use the mean of the prediction errors of all training samples as the penalty coefficient for the high-frequency band. Since most environmental noise in real-world applications exists in the high-frequency band, it is only necessary to obtain the penalty coefficient for the high-frequency band.

[0046] Based on the penalty coefficient of the high frequency band, the discrimination weight of each element in each group in the high frequency band is weighted to update the discrimination weight of each element in each group in each frequency band. Among them, the penalty coefficient for the high-frequency band ranges from 0 to 1. After weighting the discrimination weight in the high-frequency band, it is necessary to sum and normalize the discrimination weight of each element in the three frequency bands again to update the discrimination weight.

[0047] The penalty coefficient, discrimination weight, comprehensive feature vector and training sample set of high-frequency bands are updated iteratively to optimize the pulse recognition model until the preset number of iterations is reached and the optimization training stops. Specifically, based on the updated discrimination weights, the second feature vector of the amplitude sequence relative to each element in its group is updated synchronously, and then the comprehensive feature vector of each amplitude sequence is regenerated to update the training sample set; based on the updated training sample set, the pulse recognition model is optimized and trained; the above update and optimization process is repeated to achieve iterative optimization of the pulse recognition model; in addition, data records of newly diagnosed patients within a certain period of time can be added to the sample library as new samples for the pulse recognition model, thereby periodically updating the pulse recognition model.

[0048] Step S6: Collect pulse data from patients to be diagnosed and output pulse labels based on the pulse recognition model.

[0049] Specifically, based on the data acquisition process in step S1, the pulse signals of the corresponding parts of the left and right hands of the patient to be diagnosed are collected by the intelligent pulse diagnostic instrument to obtain six amplitude sequences corresponding to the patient to be diagnosed, and they are classified into the corresponding groups respectively. Fourier transform is performed on the six amplitude sequences to obtain the spectrum corresponding to each amplitude sequence. Based on the attention weight of each frequency band in each group, the first feature vector corresponding to the six amplitude sequences is obtained. Based on the discrimination weight of each element in each group in each frequency band obtained from the last iteration update, the second feature vector corresponding to the six amplitude sequences is obtained, and then the comprehensive feature vector corresponding to the six amplitude sequences is generated. Based on hand features, the three comprehensive feature vectors corresponding to the left and right hands of the patient to be diagnosed are sequentially input into the pulse recognition model, and the output pulse labels corresponding to the left and right hands of the patient to be diagnosed are output respectively. The pulse labels output by the pulse recognition model need to be interpreted by a professional physician and examined manually by a professional physician.

[0050] In addition, the pulse diagnosis results of professional physicians can be reviewed and judged by calculating the Kappa coefficient between the pulse label of the patient to be diagnosed and the actual pulse as determined by the professional physician. A judgment threshold is set. When the Kappa coefficient is less than the preset judgment threshold, the pulse is marked as doubtful and needs to be reviewed. It needs to be returned to the professional physician for a second pulse diagnosis review, thereby effectively reducing the probability of errors and omissions in manual pulse diagnosis.

[0051] The flowchart provided in this embodiment is not intended to indicate that the operations of the method will be performed in any particular order, or that all operations of the method are included in every case. Furthermore, the method may include additional operations. Within the scope of the technical concept provided by the method in this embodiment, additional variations can be made to the above method.

[0052] It should be understood that in some embodiments, the components may be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods may be implemented using software or firmware stored in memory and executed by a suitable instruction execution system.

[0053] This invention also discloses an intelligent analysis system that integrates multimodal clinical data, used to implement the aforementioned intelligent analysis method for integrating multimodal clinical data. The system structure is as follows: Figure 2 As shown, it includes: a processor, a memory, a communication interface, a data acquisition module and a display module in the intelligent pulse diagnosis instrument. The processor stores computer program instructions for implementing the above-mentioned intelligent analysis method that integrates multimodal clinical data. The communication interface is electrically connected to the data acquisition module and the display module in the intelligent pulse diagnosis instrument.

[0054] The computer program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, status setting data, integrated circuit configuration data, or source code or object code written in any combination of one or more programming languages ​​and procedural programming languages.

[0055] Computer program instructions can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server.

[0056] In the latter case, the remote computer can connect to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can connect to an external computer, such as via the Internet using an Internet service provider.

[0057] In some embodiments, in order to perform aspects of the present invention, electronic circuits including, for example, programmable logic circuits, field-programmable gate arrays (FPGAs) or programmable logic arrays (PLAs) can execute computer-readable program instructions to personalize the electronic circuits by utilizing state information of computer-readable program instructions.

[0058] The embodiments included in this invention are descriptions of preferred embodiments of the invention and are not limited to the precise structures already described above and shown in the accompanying drawings. Various modifications and changes can be made without departing from the scope of protection. All variations and improvements made by those skilled in the art to the technical solutions of this invention without departing from the design concept of this invention should fall within the scope of protection of this invention.

Claims

1. An intelligent analysis method integrating multimodal clinical data, characterized in that, include: Amplitude sequences, hand features, and pulse labels of confirmed patients under different applied pressure levels were obtained, and the amplitude sequences were grouped based on hand features and applied pressure levels. Adaptive frequency band division is performed on the spectrum of the amplitude sequence within each group to obtain the optimal frequency band segmentation scheme for each group, and attention weights for each frequency band in each group are obtained based on the optimal frequency band segmentation results. Based on the eight elements of pulse and pulse type, the amplitude sequence within each group is subgrouped, and the differences between subgroups under each element within the same group in each frequency band are calculated to obtain the discrimination weight of each element in each group in each frequency band. The amplitude values ​​of each frequency in the spectrum of the amplitude sequence are weighted based on attention weight to obtain the first feature vector of the amplitude sequence. The amplitude values ​​of each frequency in the spectrum of the amplitude sequence are weighted based on discrimination weight to obtain the second feature vector of the amplitude sequence relative to each element in its group. The first feature vector and the second feature vector are combined to generate the comprehensive feature vector of each amplitude sequence. A pulse recognition model is constructed and a training sample set is built based on the comprehensive feature vector. The error between the model prediction result and the real pulse label is calculated during the training process to construct a discrimination weight penalty mechanism and iteratively optimize the model. Collect pulse data from patients awaiting pulse diagnosis and output pulse labels based on the pulse recognition model.

2. The intelligent analysis method for fusing multimodal clinical data according to claim 1, characterized in that, The process of acquiring amplitude sequences, hand features, and pulse labels of confirmed patients under different applied pressure levels, and grouping the amplitude sequences based on hand features and applied pressure levels, includes: The pulse signals of the patient's left and right hands were collected by an intelligent pulse diagnostic instrument at the same sampling frequency. The pulse signals were collected once each under three pressure levels: superficial, medium, and deep. The collected pulse signals were standardized to obtain six equal-length amplitude sequences. The historical data of multiple diagnosed patients were obtained from the history of the intelligent pulse diagnosis instrument. Each diagnosed patient's historical data contained three amplitude sequences corresponding to the left and right hands under three applied pressure levels, and a pulse label. All amplitude sequences were grouped according to hand characteristics and applied pressure levels, resulting in six groups: left-hand float, left-hand middle pick, left-hand sink, right-hand float, right-hand middle pick, and right-hand sink.

3. The intelligent analysis method for fusing multimodal clinical data according to claim 1, characterized in that, The process of obtaining the attention weights for each frequency band in each group based on the optimal frequency band segmentation results includes: Select any group as the target group, perform Fourier transform on the amplitude sequence within the target group, and obtain the spectrum diagram corresponding to each amplitude sequence within the target group; A candidate set of preset frequency band segmentation schemes is selected, and any frequency band segmentation scheme within the set is selected as the target segmentation scheme. Based on the target segmentation scheme, the frequency range of each spectrogram is divided into three frequency bands: low, medium, and high. The standard deviation of the amplitude of each spectrogram within the target group in the same frequency band is calculated, and the mean of the standard deviation is used as the information richness of the target group in that frequency band. Similarly, the information richness of the target group in each frequency band is obtained. The cumulative value of the information richness of the target group in each frequency band is used as the evaluation score of the target segmentation scheme relative to the target group.

4. The intelligent analysis method for fusing multimodal clinical data according to claim 3, characterized in that, The step of obtaining the attention weights for each frequency band in each group based on the optimal frequency band segmentation results also includes: Traverse the candidate set of frequency band partitioning schemes to obtain the evaluation score of each frequency band partitioning scheme relative to the target group. Select the frequency band partitioning scheme with the highest evaluation score as the optimal frequency band partitioning scheme for the target group. Similarly, obtain the optimal frequency band partitioning scheme for each group. Based on the optimal frequency band segmentation scheme of the target group, the frequency range of each spectrum map within the target group is divided to obtain the optimal frequency band segmentation result of the target group; Based on the optimal frequency band segmentation results of the target group, the optimal information richness of the target group in each frequency band is obtained. The normalized optimal information richness of the target group in each frequency band is used as the attention weight of each frequency band in the target group. Similarly, the attention weight of each frequency band in each group is obtained.

5. The intelligent analysis method for fusing multimodal clinical data according to claim 1, characterized in that, The process of obtaining the discrimination weight of each element in each group in each frequency band includes: Based on the eight elements of pulse diagnosis and the twenty-eight types of pulse diagnosis, each group is divided into eight major groups corresponding to the eight elements of pulse diagnosis and 28 subgroups corresponding to the twenty-eight types of pulse diagnosis. Each major group contains multiple subgroups. Each amplitude sequence within the same group corresponds to a pulse type label. Amplitude sequences with the same pulse type label within the same group are classified into the corresponding subgroups. By traversing each group, multiple subgroups under each element within each group are obtained. Select any one of the eight elements of pulse diagnosis as the target element, extract any two subgroups under the target element in the same group, arbitrarily select one amplitude sequence from each of the two subgroups, and calculate the root mean square error of the amplitude between the spectra of the two amplitude sequences in the same frequency band.

6. The intelligent analysis method for fusing multimodal clinical data according to claim 5, characterized in that, The step of obtaining the discrimination weight of each element in each group in each frequency band also includes: Traverse the two subgroups, obtain the root mean square error of the amplitude values ​​of all amplitude sequences in the two subgroups within the same frequency band, and use the mean of the root mean square error as the difference between the two subgroups in that frequency band. The mean value of the differences between subgroups of the target element within the same group within the same frequency band is used as the distinguishing characteristic value of the target element in that group in that frequency band. Similarly, the distinguishing characteristic values ​​of the target element in each frequency band within the group are obtained. The normalized distinguishability feature value of the target element in each frequency band within the same group is used as the distinguishability weight of the target element in each frequency band within the corresponding group. By traversing each element in each group, the distinguishability weight of each element in each group within each frequency band is obtained.

7. The intelligent analysis method for fusing multimodal clinical data according to claim 1, characterized in that, The generation of the comprehensive feature vector for each amplitude sequence includes: Based on the attention weights of each frequency band in the group, the amplitudes of each frequency within the corresponding frequency band range in the spectrum of each amplitude sequence in the group are weighted to obtain the first feature vector of each amplitude sequence in the group; Based on the discrimination weight of each element in each frequency band within the group, the amplitude of each frequency within the corresponding frequency band range in the spectrum of each amplitude sequence in the group is weighted to obtain the second feature vector of the amplitude sequence relative to each element in its group. Each amplitude sequence corresponds to a first feature vector and eight second feature vectors. The first feature vector and eight second feature vectors of each amplitude sequence are concatenated in the same order to obtain the comprehensive feature vector of each amplitude sequence.

8. The intelligent analysis method for fusing multimodal clinical data according to any one of claims 1 to 7, characterized in that, The construction of the discrimination weight penalty mechanism for iterative optimization of the model includes: A pulse recognition model was constructed using a multilayer perceptron structure, including an input layer, three hidden fully connected layers, and an output layer. Based on hand features, the comprehensive feature vectors and corresponding pulse labels of the left and right hands under different applied pressure levels were used as training samples, with two training samples for each confirmed patient. The training sample set was obtained by iterating through all confirmed patients. During the training process of the model, the processing result of the third hidden fully connected layer is the predicted probability of the occurrence of each of the twenty-eight pulse types. For any training sample, extract the actual probability of each pulse type from its pulse label, calculate the absolute difference between the actual probability and the predicted probability of each pulse type, and use the mean of the absolute difference as the prediction error of the training sample. Iterate through the training sample set and use the mean of the prediction errors of all training samples as the penalty coefficient for the high-frequency band. Based on the penalty coefficient of the high frequency band, the discrimination weight of each element in each group in the high frequency band is weighted to update the discrimination weight of each element in each group in each frequency band. The penalty coefficient, discrimination weight, comprehensive feature vector, and training sample set of the high-frequency band are iteratively updated to optimize the pulse recognition model until the preset number of iterations is reached and the optimization training stops.

9. The intelligent analysis method for fusing multimodal clinical data according to claim 8, characterized in that, The process of collecting pulse data from patients awaiting pulse diagnosis and outputting pulse labels based on a pulse recognition model includes: The intelligent pulse diagnosis instrument collects pulse signals from corresponding parts of the left and right hands of the patient to be diagnosed, and obtains six amplitude sequences corresponding to the patient's pulse, which are then classified into the corresponding groups. Fourier transform is performed on the six amplitude sequences to obtain the spectrum corresponding to each amplitude sequence. Based on the attention weight of each frequency band in each group, the first feature vector corresponding to the six amplitude sequences is obtained. Based on the discrimination weight of each element in each group in each frequency band obtained from the last iteration update, the second feature vector corresponding to the six amplitude sequences is obtained, and then the comprehensive feature vector corresponding to the six amplitude sequences is generated. Based on hand-specific features, the three comprehensive feature vectors corresponding to the left and right hands of the patient to be diagnosed are sequentially input into the pulse recognition model, and the output pulse labels corresponding to the left and right hands of the patient to be diagnosed are respectively.

10. An intelligent analysis system integrating multimodal clinical data, characterized in that, include: The device includes a processor, a memory, a communication interface, a data acquisition module, and a display module. The processor stores computer program instructions for implementing the intelligent analysis method for fusing multimodal clinical data as described in any one of claims 1 to 9. The communication interface is electrically connected to the data acquisition module and the display module in the intelligent pulse diagnostic instrument.

Images