Vocal cord lesion detection method and system based on multi-modal information fusion

Abstract

The invention provides a vocal cord lesion detection method based on multi-modal information fusion, and the method comprises the following steps: S1, collecting vocal cord image data and a sound signal, and carrying out the preprocessing of the vocal cord image data and the sound signal; s2, constructing two same improved Resnet models, namely an improved Resnet model I and an improved Resnet model II; s3, determining sound features according to the preprocessed sound signals, and inputting the preprocessed vocal cord image data into the trained improved Resnet model I to obtain vocal cord image features; s4, fusing the vocal cord image features and the sound features to obtain multi-modal fusion features; and S5, inputting the multi-modal fusion features into the trained improved Resnet model II to obtain predicted vocal cord lesion information. Through the method, the diagnosis time can be shortened, and the diagnosis accuracy can be improved.

Description

Technical Field

[0001] The present invention relates to the field of medical analysis, and in particular to a vocal cord lesion detection method and system based on multimodal information fusion. Background Art

[0002] Vocal cord lesions refer to a series of abnormal changes or pathological conditions in the vocal cord tissue, including vocal cord polyps, nodules, leukoplakia, and laryngeal cancer. At present, the diagnosis of laryngeal diseases mainly relies on experienced otolaryngologists, who make judgments through laryngoscopy, acoustic analysis, and other methods. However, this traditional diagnostic method is highly dependent on the doctor's professional knowledge and experience, and is subjective. On the other hand, the inspection and analysis process is complicated and may require multiple follow-up visits, which increases the burden on patients. At the same time, this method is not very accurate. Due to the limitations of human eye recognition, some minor lesions may be difficult to detect, leading to misdiagnosis or missed diagnosis.

[0003] Therefore, it is urgent to propose a new technical means to solve the above technical problems. Summary of the invention

[0004] In view of this, in order to improve the accuracy of diagnosis, the present invention proposes a vocal cord lesion detection method and system based on multimodal information fusion.

[0005] The present invention provides a method for detecting vocal cord lesions based on multimodal information fusion, comprising the following steps:

[0006] S1. Collecting vocal cord image data and sound signals, and preprocessing the vocal cord image data and sound signals;

[0007] S2. Construct two identical improved Resnet models, namely improved Resnet model I and improved Resnet model II;

[0008] S3. Determine the sound features according to the preprocessed sound signal, and input the preprocessed vocal cord image data into the trained improved Resnet model I to obtain the vocal cord image features;

[0009] S4. fusing the vocal cord image features and the sound features to obtain a multimodal fusion feature;

[0010] S5. Input the multimodal fusion features into the trained improved Resnet model II to obtain the predicted vocal cord lesion information.

[0011] Furthermore, the multimodal fusion feature is obtained through the following steps:

[0012] The vocal cord image features and sound features are standardized, and the standardized vocal cord image features and sound features are spliced ​​together using the feature-level fusion method to obtain the multimodal fusion feature F fusion , multimodal fusion feature F fusion It is expressed by the following formula:

[0013] F fusion =[F image ,F audio ]

[0014] Among them, F image represents the standardized vocal cord image features, F audio Represents the normalized sound characteristics.

[0015] Furthermore, when the multimodal fusion feature F fusion Before inputting into the improved Resnet model II, the high-dimensional multimodal fusion feature F is transformed by principal component analysis. fusion Project it into a low-dimensional space, filter out features through recursive feature elimination, and input the retained features into the trained improved Resnet model II for prediction.

[0016] Further, the sound characteristics are determined by the following method:

[0017] Extract time domain features, frequency domain features and time-frequency features from the preprocessed sound signal, combine the features contained in the time domain features, frequency domain features and time-frequency features in pairs, calculate the Spearman correlation coefficient of each combination, and determine whether the Spearman correlation coefficient is higher than a set threshold. If so, remove any one feature in the combination and retain the remaining features; if not, retain both features in the combination; combine the retained features into a vector and reduce the feature space dimension through principal component analysis to obtain sound features;

[0018] When the Spearman correlation coefficients of multiple combinations are all higher than the set threshold and there are repeated features in multiple combinations, the repeated features are retained and the other feature in the combination is eliminated.

[0019] Further, the time domain features include mean, variance, skewness and kurtosis of the sound signal;

[0020] The frequency domain characteristics include energy distribution, spectrum center and spectrum bandwidth of the sound signal;

[0021] The time-frequency feature is extracted by combining the time domain feature and the frequency domain feature with the Mel-frequency cepstral coefficient.

[0022] Further, the preprocessing of the sound signal includes:

[0023] Use low-pass filtering to remove high-frequency noise, use amplitude normalization to adjust the amplitude of the sound signal, use detrending to remove linear or nonlinear trends in the signal, and use downsampling to reduce the sampling rate of the signal.

[0024] Furthermore, the improved Resnet model replaces all residual modules in the Resnet model with improved residual modules;

[0025] The improved residual module is to sequentially connect an efficient channel attention module and a spatial channel attention module after the last convolution module in the residual module.

[0026] Furthermore, the loss function LOSS is used to train the improved Resnet model:

[0027]

[0028] Where N is the number of samples, n is the sample number, I is the number of categories, i is the category number, and y is the sample number. n,i represents the one-hot encoding of the true label of the nth sample, p n,i Represents the predicted probability of category i of the nth sample by the improved Resnet model.

[0029] Further, the preprocessing of the vocal cord image data includes:

[0030] Gaussian filtering is used to denoise the image, histogram equalization is used to enhance the contrast, and the Canny algorithm is used to detect image edges and highlight the contours of the lesion area.

[0031] Accordingly, a system for diagnosing vocal cord lesions based on the above multimodal information fusion is also provided, comprising a data acquisition module, a data processing module and a prediction module;

[0032] The acquisition module is used to acquire vocal cord image data and sound signals, and transmit the acquired vocal cord image data and sound signals to the data processing module;

[0033] The data processing module is used to preprocess the collected vocal cord image data and sound signals, and determine the vocal cord image features and sound features according to the preprocessed vocal cord image data and sound signals; and perform feature fusion on the vocal cord image features and sound features to obtain multimodal fusion features and transmit them to the prediction module;

[0034] The prediction module performs prediction based on the input multimodal fusion features.

[0035] Beneficial effects of the present invention: The present invention obtains multimodal fusion features by fusing image and sound signals, which can more comprehensively describe the complex characteristics of vocal cord lesions; inputting the multimodal fusion features into the improved Resnet model for diagnostic prediction can reduce the diagnosis time and improve the accuracy of diagnosis. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The present invention will be further described below in conjunction with the accompanying drawings and embodiments:

[0037] Figure 1 It is a flow chart of the present invention.

[0038] Figure 2 This is a schematic diagram of the improved Resnet model structure of the present invention.

[0039] Figure 3 Schematic diagram of the system structure of the present invention.

[0040] Figure 4 A plan view showing the multimodal vocal cord lesion automatic detection system of the present invention. DETAILED DESCRIPTION

[0041] The present invention is further described below in conjunction with the accompanying drawings:

[0042] The present invention provides a method for detecting vocal cord lesions based on multimodal information fusion, comprising the following steps:

[0043] S1. Collecting vocal cord image data and sound signals, and preprocessing the vocal cord image data and sound signals;

[0044] S2. Construct two identical improved Resnet models, namely improved Resnet model I and improved Resnet model II;

[0045] S3. Determine the sound features according to the preprocessed sound signal, and input the preprocessed vocal cord image data into the trained improved Resnet model I to obtain the vocal cord image features;

[0046] S4. fusing the vocal cord image features and the sound features to obtain a multimodal fusion feature;

[0047] S5. Input the multimodal fusion features into the trained improved Resnet model II to obtain the predicted vocal cord lesion information. Through the above method, errors caused by subjective diagnosis can be avoided and the accuracy of diagnosis can be improved.

[0048] In this embodiment, in step S1, an Olympus OTV-S190 system is used to connect a high-resolution electronic fiber laryngoscope to the patient's larynx to collect vocal cord image data; a RodeNT1-A dedicated microphone is used to collect the patient's voice signal.

[0049] Preprocessing vocal cord image data and sound signals;

[0050] In this embodiment, the preprocessing of the sound signal includes:

[0051] Use low-pass filtering to remove high-frequency noise, use amplitude normalization to adjust the amplitude of the sound signal, use detrending to remove linear or nonlinear trends in the signal, and use downsampling to reduce the sampling rate of the signal; the above preprocessing methods can be performed in any order, and are preferably performed in the above order;

[0052] The expression for removing high-frequency noise using low-pass filtering is:

[0053]

[0054] Among them, y[n] represents the filtered signal, k represents the index variable in the summation operation, which is used to traverse all values ​​from 0 to M-1 and has no actual physical meaning. h[k] represents the impulse response coefficient of the low-pass filter, and x[nk] represents the nkth sample value of the input signal, that is, the value of the input signal after k samples are delayed. M represents the length of the low-pass filter.

[0055] The expression for adjusting the amplitude of the sound signal using amplitude normalization is:

[0056]

[0057] Among them, x norm [n] represents the normalized signal, x[n] represents the original signal, and max(x[n]) represents the maximum absolute value of the original signal. The above method can ensure that the signal is within a standard range.

[0058] The expression for removing linear or nonlinear trends in a signal using detrending is:

[0059] x detrend [n] = x[n] - (a·n+b)

[0060] Among them, x detrend [n] represents the detrended signal, x[n] represents the original signal, and a and b represent the trend line coefficients estimated by linear regression. Detrending the signal allows for better analysis of the volatility component.

[0061] The expression for reducing the sampling rate of a signal using downsampling is:

[0062] y[m]=x[Dm]

[0063] Among them, y[m] represents the downsampled signal, x represents the original signal, D represents the downsampling factor, and m represents the position of the sampling point of the downsampled signal.

[0064] The original signal in each preprocessing method refers to the information input into the formula, which is the signal output by the previous preprocessing method. For example, the preprocessing steps of the sound signal are filtering, normalization, detrending and downsampling. Then the original signal input into the normalization formula is the signal output by filtering, and the original signal input into the detrending formula is the signal output by normalization, and so on.

[0065] In this embodiment, the preprocessing of the vocal cord image data includes:

[0066] Gaussian filtering is used to denoise the image, histogram equalization is used to enhance the contrast, and the Canny algorithm is used to detect image edges and highlight the contours of the lesion area.

[0067] The expression for image denoising using Gaussian filtering is:

[0068]

[0069] Among them, G(i,j) represents the element value of the Gaussian filter, σ represents the standard deviation of the Gaussian distribution, x and y represent the horizontal and vertical coordinates of the pixel respectively, i and j represent the offset relative to the center of the Gaussian kernel, d represents half of the filter window size, and I filtered represents the filtered image, and I represents the original image. The above method can reduce image noise and improve image clarity and quality.

[0070] The process of using histogram equalization to enhance contrast and using the canny algorithm to detect image edges is an existing technology and will not be described in detail here. Through the above method, the difference between the object and the background in the image can be improved, making the lesion area easier to identify.

[0071] In this embodiment, in step S2, two identical improved Resnet models are constructed, namely, improved Resnet model I and improved Resnet model II;

[0072] The improved Resnet model replaces all residual modules in the Resnet model with improved residual modules;

[0073] The improved residual module is to connect the efficient channel attention module (ECA) and the spatial channel attention module (CBAM) in sequence after the last convolution module (conv) in the residual module, such as Figure 2 As shown; Figure 2 Where 1×1×C represents the feature map size, GAP (Global Average Pooling) represents global average pooling, K=Φ(C) represents the convolution kernel size K determined by the channel dimension C, δ represents the sigmoid function, Attention mechanism represents the attention mechanism, relu represents the activation function, residual module represents the residual module, that is, the residual module in the original resnet, EC-residual module represents the residual module with the efficient channel attention module and the spatial channel attention module added, and EC-resnet represents the improved resnet model.

[0074] Among them, the residual module, efficient channel attention module, spatial channel attention module and resnet model are all existing technologies, and their structures are not described here.

[0075] The efficient channel attention module can improve the expressiveness and discrimination of features by enhancing the correlation between different channels; the spatial channel attention module combines channel attention (Channel Attention Mechanism) and spatial attention (Spatial Attention Mechanism), which can specifically enhance the representation of important features and suppress the interference of irrelevant information. The above models can more accurately capture key lesion features, improve diagnostic accuracy and model stability, and thus provide reliable data support for clinical practice.

[0076] In this embodiment, in step S3, the sound features are determined according to the preprocessed sound signal, and the preprocessed vocal cord image data is input into the trained improved Resnet model I to obtain the vocal cord image features;

[0077] The process of determining the sound characteristics based on the preprocessed sound signal is as follows:

[0078] Extract time domain features, frequency domain features and time-frequency features from the preprocessed sound signal, combine the features contained in the time domain features, frequency domain features and time-frequency features in pairs, calculate the Spearman correlation coefficient of each combination, and determine whether the Spearman correlation coefficient is higher than a set threshold. If yes, remove any one feature in the combination and retain the remaining features; if not, retain both features in the combination; combine the retained features into a vector and reduce the feature space dimension by principal component analysis to obtain sound features; the process of reducing the feature space dimension by principal component analysis and the process of calculating the Spearman correlation coefficient of two features are prior arts and will not be described in detail here;

[0079] When the Spearman correlation coefficients of multiple combinations are all higher than the set threshold, the threshold is set according to the requirements, preferably 0.8, and there are repeated features in multiple combinations, the repeated features are retained and the other features in the combination are eliminated; for example, there are two combinations whose Spearman correlation coefficients are higher than the set threshold, and the two groups of features are A and B, and B and C; at this time, the repeated feature B is retained, and A and C in the two groups are eliminated respectively;

[0080] Specifically, the time domain features include mean, variance, skewness and kurtosis based on sound waveform statistics; the frequency domain features include the energy distribution, spectrum center and spectrum bandwidth of the sound signal; the time-frequency features combine the time domain features and frequency domain features with the Mel-frequency cepstral coefficients to extract the time-frequency features. The process of extracting the time-frequency features using the Mel-frequency cepstral coefficients is an existing technology and will not be described here. The above method can select the most relevant features, avoid unnecessary redundant information, and can reduce the dimension of the feature space through principal component analysis, thereby improving the computational efficiency and generalization ability of the model.

[0081] The process of determining vocal cord characteristics is as follows:

[0082] The improved Resnet model I is trained until convergence, and the training is completed. The batch_size of the number of data (samples) passed to the program for training is set to 32. The number of epochs of the complete traversal of the data set during training is 200. The optimizer is Adam, the learning rate is 0.0001, and the loss function LOSS is used for training. The formula of the loss function LOSS is as follows:

[0083]

[0084] Where N is the number of samples, n is the sample number, I is the number of categories, i is the category number, and y is the sample number. n,i represents the one-hot encoding of the true label of the nth sample, p n,i Represents the predicted probability of category i of the nth sample by the improved Resnet model.

[0085] The preprocessed image data is input into the trained improved Resnet model I to obtain the vocal cord image features.

[0086] In this embodiment, in step S4, the vocal cord image feature and the sound feature are fused to obtain a multimodal fusion feature;

[0087] The multimodal fusion features are obtained through the following steps:

[0088] Standardize the vocal cord image features and sound features. Both the vocal cord image features and sound features are vectors. The process of standardizing the vectors is a prior art and will not be described in detail here.

[0089] The feature-level fusion method is used to combine the standardized vocal cord image features and sound features to obtain the multimodal fusion feature F fusion , multimodal fusion feature F fusion It is expressed by the following formula:

[0090] F fusion =[F image ,F audio ]

[0091] Among them, F image represents the standardized vocal cord image features, F audio Represents the normalized sound characteristics.

[0092] In the multimodal fusion feature F fusion Before inputting into the improved Resnet model II, the high-dimensional multimodal fusion feature F is transformed by principal component analysis. fusion Projecting to a low-dimensional space, and filtering out features by recursive feature elimination, and inputting the retained features into the trained improved Resnet model II for prediction; wherein, using principal component analysis to project high-dimensional feature vectors to a low-dimensional space is an existing technology, and using recursive feature elimination to filter out features is also an existing technology, and the specific process will not be described in detail here;

[0093] The improved Resnet model II is trained until convergence, and the training is completed. The batch_size of the number of data (samples) passed to the program for training is set to 32. The number of epochs of the complete traversal of the data set during training is 200. The optimizer is Adam, the learning rate is 0.0001, and the loss function LOSS is used for training. The loss function LOSS formula is as follows:

[0094]

[0095] Where N is the number of samples, n is the sample number, I is the number of categories, i is the category number, and y is the sample number. n,irepresents the one-hot encoding of the true label of the nth sample, p n,i Represents the predicted probability of category i of the nth sample by the improved Resnet model.

[0096] Using the same model to extract vocal cord image features and make predictions can optimize features and improve model performance; combining image features and acoustic features to obtain multimodal fusion features can more comprehensively describe the complex features of vocal cord lesions and help improve the accuracy of model diagnosis; and projecting multimodal fusion features into a low-dimensional space can remove redundant and irrelevant features while retaining as much original information as possible, thereby improving the generalization ability of the model; and using the recursive feature elimination method to screen features can gradually remove unimportant features and retain the most representative features, thereby reducing the complexity of the model.

[0097] Accordingly, a system for diagnosing vocal cord lesions based on the above multimodal information fusion is also provided, comprising a data acquisition module, a data processing module and a prediction module;

[0098] The acquisition module is used to acquire vocal cord image data and sound signals, and transmit the acquired vocal cord image data and sound signals to the data processing module;

[0099] The data processing module is used to preprocess the collected vocal cord image data and sound signals, and determine the vocal cord image features and sound features according to the preprocessed vocal cord image data and sound signals; and perform feature fusion on the vocal cord image features and sound features to obtain multimodal fusion features and transmit them to the prediction module;

[0100] In this embodiment, before transmitting the multimodal fusion features to the prediction module, the multimodal fusion features are processed by principal component analysis and recursive feature elimination in sequence;

[0101] The prediction module makes predictions based on input multimodal fusion features or features processed by recursive feature elimination.

[0102] In this embodiment, a display device can also be provided for displaying the collected data, various preprocessing steps, extracted vocal cord image features and sound features (i.e. Figure 4 The image features and acoustic features in ), as well as the multimodal fusion features and prediction results (i.e. Figure 4 fusion results and diagnosis results in ).

[0103] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution of the present invention, which should be included in the scope of the claims of the present invention.

Claims

1. A method for detecting vocal cord lesions based on multimodal information fusion, characterized in that: The following steps are involved: S1. Collecting vocal cord image data and sound signals, and preprocessing the vocal cord image data and sound signals; S2. Construct two identical improved Resnet models, namely improved Resnet model I and improved Resnet model II; S3. Determine the sound features according to the preprocessed sound signal, and input the preprocessed vocal cord image data into the trained improved Resnet model I to obtain the vocal cord image features; S4. fusing the vocal cord image features and the sound features to obtain a multimodal fusion feature; S5. Input the multimodal fusion features into the trained improved Resnet model II to obtain the predicted vocal cord lesion information.

2. The method for detecting vocal cord lesions based on multimodal information fusion according to claim 1, characterized in that: The multimodal fusion features are obtained through the following steps: The vocal cord image features and sound features are standardized, and the standardized vocal cord image features and sound features are spliced ​​together using the feature-level fusion method to obtain the multimodal fusion feature F fusion , multimodal fusion feature F fusion It is expressed by the following formula: F fusion =[F image ,F audio ] Among them, F image represents the standardized vocal cord image features, F audio Represents the normalized sound characteristics.

3. The method for detecting vocal cord lesions based on multimodal information fusion according to claim 2, characterized in that: In the multimodal fusion feature F fusion Before inputting into the improved Resnet model II, the high-dimensional multimodal fusion feature F is transformed by principal component analysis. fusion Project it into a low-dimensional space, filter out features through recursive feature elimination, and input the retained features into the trained improved Resnet model II for prediction.

4. The method for detecting vocal cord lesions based on multimodal information fusion according to claim 2, characterized in that: The sound characteristics are determined by: Extract time domain features, frequency domain features and time-frequency features from the preprocessed sound signal, combine the features contained in the time domain features, frequency domain features and time-frequency features in pairs, calculate the Spearman correlation coefficient of each combination, and determine whether the Spearman correlation coefficient is higher than a set threshold. If so, remove any one feature in the combination and retain the remaining features; if not, retain both features in the combination; combine the retained features into a vector and reduce the feature space dimension through principal component analysis to obtain sound features; When the Spearman correlation coefficients of multiple combinations are all higher than the set threshold and there are repeated features in multiple combinations, the repeated features are retained and the other feature in the combination is eliminated.

5. The method for detecting vocal cord lesions based on multimodal information fusion according to claim 4, characterized in that: The time domain features include the mean, variance, skewness and kurtosis of the sound signal; The frequency domain characteristics include energy distribution, spectrum center and spectrum bandwidth of the sound signal; The time-frequency feature is extracted by combining the time domain feature and the frequency domain feature with the Mel-frequency cepstral coefficient.

6. The method for detecting vocal cord lesions based on multimodal information fusion according to claim 4, characterized in that: The preprocessing of the sound signal comprises: Use low-pass filtering to remove high-frequency noise, use amplitude normalization to adjust the amplitude of the sound signal, use detrending to remove linear or nonlinear trends in the signal, and use downsampling to reduce the sampling rate of the signal.

7. The method for detecting vocal cord lesions based on multimodal information fusion according to claim 1 or claim 3, characterized in that: The improved Resnet model replaces all residual modules in the Resnet model with improved residual modules; The improved residual module is to sequentially connect an efficient channel attention module and a spatial channel attention module after the last convolution module in the residual module.

8. The method for detecting vocal cord lesions based on multimodal information fusion according to claim 7, characterized in that: The improved Resnet model is trained using the loss function LOSS: Where N is the number of samples, n is the sample number, I is the number of categories, i is the category number, and y is the sample number. n,i represents the one-hot encoding of the true label of the nth sample, p n,i Represents the predicted probability of category i of the nth sample by the improved Resnet model.

9. The method for detecting vocal cord lesions based on multimodal information fusion according to claim 1, characterized in that: The preprocessing of the vocal cord image data comprises: Gaussian filtering is used to denoise the image, histogram equalization is used to enhance the contrast, and the Canny algorithm is used to detect image edges and highlight the contours of the lesion area.

10. A system based on the method for detecting vocal cord lesions based on multimodal information fusion according to any one of claims 1 to 9, characterized in that: It includes data acquisition module, data processing module and prediction module; The acquisition module is used to acquire vocal cord image data and sound signals, and transmit the acquired vocal cord image data and sound signals to the data processing module; The data processing module is used to preprocess the collected vocal cord image data and sound signals, and determine the vocal cord image features and sound features according to the preprocessed vocal cord image data and sound signals; and performing feature fusion on the vocal cord image features and the sound features to obtain multimodal fusion features and transmit them to the prediction module; The prediction module performs prediction based on the input multimodal fusion features.

Images