A laryngoscope image recognition method and system based on multi-feature extraction

Abstract

The application discloses a laryngoscope image recognition method and system based on multi-feature extraction, which comprises the following steps: acquiring a patient's laryngeal image and performing image data preprocessing to obtain a preprocessed patient's laryngeal image; performing multi-feature extraction processing on the preprocessed patient's laryngeal image based on a deep learning feature extraction model to obtain morphological features of the patient's laryngeal image and quantitative features of the patient's laryngeal image; performing multi-scale feature fusion processing on the morphological features of the patient's laryngeal image and the quantitative features of the patient's laryngeal image to obtain patient's laryngeal key features with feature weights; and performing human-computer interaction visualization processing on the patient's laryngeal key features with feature weights to construct a patient's laryngeal recognition diagnosis result. The embodiment of the application can perform secondary judgment on the recognized local lesion feature combination, improve the accuracy of the lesion diagnosis result and the transparency of the diagnosis. The application can be widely applied to the technical field of medical image recognition.

Description

Technical Field

[0001] The present application relates to the technical field of medical image recognition, and in particular to a laryngoscope image recognition method and system based on multi-feature extraction. Background Art

[0002] Electronic laryngoscopes are widely used to detect early-stage laryngeal cancer. Narrow-band imaging enhances the contrast of mucosal surface vessels, better characterizing tumor-specific neovascularization and detecting superficial, abnormal mucosal lesions that may be missed by conventional white-light imaging. However, significant skill differences exist among endoscopists in identifying suspicious lesions, leading to a high rate of missed diagnoses of laryngeal cancer, a serious threat to patients' lives. Improving the accuracy of endoscopic diagnosis of early-stage laryngeal cancer is of significant clinical significance.

[0003] With the continuous advancement of artificial intelligence technology, deep learning neural networks have shown great potential in assisting doctors in disease screening and diagnosis. They can not only effectively compensate for deficiencies in experience, but also significantly improve diagnostic accuracy and efficiency. Combining deep learning technology with electronic laryngoscope systems can provide more accurate and automated auxiliary diagnosis and treatment for laryngeal diseases. However, current research on AI-assisted diagnosis and treatment is mainly based on end-to-end deep learning algorithms, which only output diagnostic conclusions. The diagnostic process is an opaque and incomprehensible "black box" with no explanation of the decision-making process and diagnostic basis. This has greatly affected the credibility and acceptability of AI systems in clinical practice.

[0004] In summary, the technical problems existing in the relevant technologies need to be improved. Summary of the Invention

[0005] The main purpose of the embodiments of the present application is to propose a laryngoscope image recognition method and system based on multi-feature extraction, which can perform secondary judgment on the identified local lesion feature combination, thereby improving the accuracy of lesion diagnosis results and the transparency of diagnosis.

[0006] To achieve the above objectives, one aspect of an embodiment of the present application provides a laryngoscope image recognition method based on multi-feature extraction, the method comprising:

[0007] Acquiring a laryngeal image of the patient and performing image data preprocessing to obtain a preprocessed laryngeal image of the patient;

[0008] Performing multi-feature extraction processing on the pre-processed patient laryngeal image based on a deep learning feature extraction model to obtain morphological features and quantitative features of the patient laryngeal image;

[0009] performing multi-scale feature fusion processing on the morphological features of the patient's laryngeal image and the quantitative features of the patient's laryngeal image to obtain key features of the patient's laryngeal image with feature weights;

[0010] The key features of the patient's larynx with the feature weights are processed visually through human-computer interaction to construct a patient's larynx identification and diagnosis result.

[0011] In some embodiments, acquiring a patient laryngeal image and performing image data preprocessing to obtain a preprocessed patient laryngeal image includes:

[0012] Obtain images of the patient's larynx using an electronic laryngoscope device;

[0013] Extracting clear key frames of the patient's laryngeal image and filtering out blurry frames and non-target areas of the patient's laryngeal image to construct a coarsely screened patient's laryngeal image;

[0014] Performing accuracy testing on the coarsely screened patient laryngeal image to obtain a standard patient laryngeal image;

[0015] The standard patient laryngeal image is subjected to image enhancement and formatting processing to obtain the pre-processed patient laryngeal image.

[0016] In some embodiments, the performing accuracy testing on the coarsely screened patient laryngeal image to obtain a standard patient laryngeal image includes:

[0017] An image clarity evaluation function is constructed based on the Laplace operator, and image blur detection processing is performed on the coarsely screened patient laryngeal image to obtain a clear patient laryngeal image;

[0018] Calculating the ratio of valid matching points of the clear patient laryngeal image based on the ORB algorithm to obtain an image repeatability score;

[0019] performing an image repeatability test on the clear patient laryngeal image based on a preset scoring threshold and in combination with the image repeatability score to obtain a filtered patient laryngeal image;

[0020] The filtered patient laryngeal image is subjected to laryngeal key structural feature detection by a target detection algorithm to obtain the standard patient laryngeal image.

[0021] In some embodiments, the deep learning feature extraction model is used to perform multi-feature extraction on the pre-processed patient laryngeal image to obtain morphological features and quantitative features of the patient laryngeal image, including:

[0022] performing feature extraction processing on the preprocessed patient laryngeal image using a Transformer network model based on a cross-fusion encoder to obtain morphological features of the first patient laryngeal image, wherein the morphological features of the first patient laryngeal image include texture features, color features, boundary features, vascular morphology features, vascular color features, vascular direction features, and mucosal color features;

[0023] performing feature extraction processing on the preprocessed patient laryngeal image using a ResNet-50 deep residual network model to obtain morphological features of a second patient laryngeal image, wherein the morphological features of the second patient laryngeal image represent lesion location features;

[0024] combining the morphological features of the first patient's laryngeal image and the morphological features of the second patient's laryngeal image to obtain the morphological features of the patient's laryngeal image;

[0025] The preprocessed patient laryngeal image is subjected to lesion image analysis to obtain quantitative features of the patient laryngeal image, wherein the quantitative features of the patient laryngeal image include aspect ratio information, color spectrum information, S-channel image entropy information, texture information, directional gradient histogram information, and color moment information.

[0026] In some embodiments, the step of performing feature extraction on the preprocessed patient laryngeal image using a Transformer network model based on a cross-fusion encoder to obtain morphological features of the first patient laryngeal image includes:

[0027] Inputting the preprocessed patient laryngeal image into the Transformer network model based on the cross-fusion encoder, wherein the Transformer network model based on the cross-fusion encoder includes a plurality of image feature extraction modules and linear layers, wherein the plurality of image feature extraction modules include a patch embedding layer and a cross-fusion encoder module;

[0028] performing dimensionality reduction processing on the preprocessed patient laryngeal image based on a patch embedding layer of a Transformer network model of the cross-fusion encoder to obtain a dimensionality-reduced patient laryngeal image;

[0029] performing image feature extraction processing on the dimensionality-reduced patient laryngeal image based on a cross-fusion encoder module of a Transformer network model of the cross-fusion encoder to obtain preliminary morphological features of the first patient laryngeal image;

[0030] Based on the linear layer of the Transformer network model of the cross-fusion encoder, image feature category prediction processing is performed on the morphological features of the preliminary first patient laryngeal image to obtain the morphological features of the first patient laryngeal image.

[0031] In some embodiments, the cross-fusion encoder module of the Transformer network model based on the cross-fusion encoder performs image feature extraction processing on the dimensionally reduced patient laryngeal image to obtain preliminary morphological features of the first patient laryngeal image, including:

[0032] Inputting the dimensionally reduced patient laryngeal image into a cross-fusion encoder module of a Transformer network model of the cross-fusion encoder, wherein the cross-fusion encoder module includes a first normalization layer, a parallel attention layer, a second normalization layer, and a multilayer perceptron;

[0033] Based on the first normalization layer and the second normalization layer of the cross-fusion encoder module, feature extraction is performed on the dimensionally reduced patient laryngeal image to obtain a patient laryngeal feature image;

[0034] Based on the parallel attention layer of the cross-fusion encoder module, the patient's laryngeal feature image is divided into global feature information and fine-grained features and cross-splicing is performed to obtain a spliced ​​patient laryngeal image feature;

[0035] Based on the multi-layer perceptron of the cross-fusion encoder module, the features of the spliced ​​patient laryngeal image are perceived and output to obtain the morphological features of the preliminary first patient laryngeal image.

[0036] In some embodiments, the multi-scale feature fusion processing is performed on the morphological features of the patient's laryngeal image and the quantitative features of the patient's laryngeal image to obtain the key features of the patient's laryngeal image with feature weights, including:

[0037] fusing the morphological features of the patient's laryngeal image with the quantitative features of the patient's laryngeal image to obtain multi-scale feature data of the patient's laryngeal image;

[0038] Based on the random forest model, a sampling method with replacement was used to repeatedly randomly select multi-scale feature data of the patient's laryngeal images to construct a training sample subset.

[0039] Training multiple decision trees based on the training sample subset, each tree randomly selecting features to split during the training process, and constructing a random forest model;

[0040] Voting and decision-making are performed on the multi-scale feature data of the patient's laryngeal image based on the random forest model to obtain the key features of the patient's laryngeal image with feature weights.

[0041] In some embodiments, the voting decision expression of the random forest model is specifically as follows:

[0042]

[0043] In the above formula, Represents the final classification result, T m (x) represents the classification result of the mth tree for sample x, C represents the number of sample categories, c represents the sample category, and ∥(·) represents the indicator function.

[0044] In some embodiments, performing human-computer interactive visualization processing on the key features of the patient's larynx with the feature weights to construct the patient's larynx identification and diagnosis results includes:

[0045] Outputting interpretable information based on the key features of the patient's larynx with the feature weights, the interpretable information including texture information with weighted proportions, tumor boundary information with weighted proportions, vascular morphology information with weighted proportions, S-channel image entropy information with weighted proportions, vascular color information with weighted proportions, and vascular direction information with weighted proportions;

[0046] Performing visual significance labeling processing on the key features of the patient's larynx with the feature weights according to the interpretable information to obtain the lesion area of ​​the patient's larynx;

[0047] Confirmation feedback is performed on the lesion area of ​​the patient's throat to obtain the patient's throat identification diagnosis result.

[0048] To achieve the above objectives, another aspect of the present application provides a laryngoscope image recognition system based on multi-feature extraction, the system comprising:

[0049] The first module is used to obtain a laryngeal image of a patient and perform image data preprocessing to obtain a preprocessed laryngeal image of the patient;

[0050] The second module is used to perform multi-feature extraction processing on the pre-processed patient laryngeal image based on a deep learning feature extraction model to obtain morphological features and quantitative features of the patient laryngeal image;

[0051] A third module is configured to perform multi-scale feature fusion processing on the morphological features of the patient's laryngeal image and the quantitative features of the patient's laryngeal image to obtain key features of the patient's laryngeal image with feature weights;

[0052] The fourth module is used to perform human-computer interactive visualization processing on the key features of the patient's larynx with feature weights to construct the patient's larynx identification and diagnosis results.

[0053] The embodiments of the present application include at least the following beneficial effects: The present application provides a laryngoscope image recognition method and system based on multi-feature extraction, which obtains the patient's laryngeal image and performs image data preprocessing to extract the local lesion area that meets the clinical diagnosis, and further performs multi-feature extraction processing through a deep learning feature extraction model to obtain the morphological features and quantitative features of the patient's laryngeal image, and then performs multi-scale feature fusion processing to output the key features of the patient's laryngeal image with feature weights. By performing a secondary judgment on the identified local lesion feature combination, it can guide clinical decision-making, obtain more accurate lesion diagnosis results, reduce the misdiagnosis rate, and finally perform human-computer interactive visualization processing, and use an explainable artificial intelligence system to diagnose early laryngeal cancer, thereby increasing the transparency of the diagnosis. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 This is a flow chart of a laryngoscope image recognition method based on multi-feature extraction provided in an embodiment of the present application;

[0055] Figure 2 1 is a schematic structural diagram of a laryngoscope image recognition system based on multi-feature extraction provided in an embodiment of the present application;

[0056] Figure 3 is a schematic diagram of image data preprocessing of a patient's laryngeal image provided by an embodiment of the present application;

[0057] Figure 4 is a schematic diagram of performing multi-feature extraction processing on a pre-processed patient laryngeal image provided by an embodiment of the present application;

[0058] Figure 5 is a schematic diagram of a multi-scale feature fusion decision provided in an embodiment of the present application;

[0059] Figure 6 This is a schematic diagram of the human-computer interaction online learning optimization provided by an embodiment of the present application. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are merely examples of systems and methods consistent with some aspects of the embodiments of the present application as detailed in the appended claims.

[0061] It will be understood that the terms "first", "second", etc. used in this application may be used herein to describe various concepts, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0062] The terms "at least one", "plurality", "each", "any", etc. used in this application include "at least one", "two" or more, "plurality" or "each", "any" or "any one", "each" or "any one" as used herein.

[0063] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0064] Reference Figure 1 , Figure 1 A flowchart of a laryngoscope image recognition method based on multi-feature extraction provided by an embodiment of the present invention, referring to Figure 1 , the method comprises the following steps:

[0065] S100, acquiring a laryngeal image of a patient and performing image data preprocessing to obtain a preprocessed laryngeal image of the patient;

[0066] Specifically, the laryngeal image is acquired in real time through the electronic laryngoscope device, clear key frames are automatically extracted, blurred frames and non-target areas are filtered out, the filtered images are further enhanced, and finally the acquired images are converted into a standardized format.

[0067] Furthermore, it should be noted that, in some embodiments, step S100 may include: S110, acquiring a patient's larynx image by an electronic laryngoscope device;

[0068] S120, extracting clear key frames of the patient's laryngeal image and filtering out blurred frames and non-target areas of the patient's laryngeal image to construct a coarsely screened patient's laryngeal image;

[0069] S130, performing accuracy testing on the coarsely screened patient laryngeal image to obtain a standard patient laryngeal image;

[0070] Furthermore, it should be noted that, in some embodiments, step S130 may include: S131, constructing an image clarity evaluation function based on the Laplace operator, performing image blur detection processing on the coarsely screened patient laryngeal image, and obtaining a clear patient laryngeal image; S132, calculating the proportion of effective matching points of the clear patient laryngeal image based on the ORB algorithm, and obtaining an image repeatability score; S133, based on a preset score threshold, performing image repeatability detection on the clear patient laryngeal image in combination with the image repeatability score, and obtaining a filtered patient laryngeal image; S134, performing laryngeal key structure feature detection on the filtered patient laryngeal image through the target detection algorithm, and obtaining a standard patient laryngeal image.

[0071] S140 , performing image enhancement and formatting processing on the standard patient laryngeal image to obtain a pre-processed patient laryngeal image.

[0072] In some specific embodiments, Figure 3 As shown, real-time laryngeal images are acquired, clear keyframes are automatically extracted, and blurred frames and non-target areas are filtered out. The filtered images are further enhanced, and finally, the acquired laryngeal images are standardized. The Laplace operator is used as the image clarity evaluation function, and the fuzzy judgment interval (a, b) is set according to quality requirements. Since larger gradient values ​​indicate clearer images, only images with a gradient value greater than b are retained. An image reproducibility score—the percentage of valid matching points—is calculated based on the ORB (Oriented FAST and Rotated BRIEF) algorithm. 0 indicates no valid matching points (completely non-repeated), while 1 indicates all matching points are validated (completely repetitiv). A threshold is set based on the score; values ​​above 0.6 indicate significant repetitive areas, which are then filtered out. An object detection algorithm is also used to detect the presence of key laryngeal structures, ensuring that the target area is included in the image. Finally, the processed images are denoised, brightness adjusted, and contrast enhanced. The images are then normalized to a uniform size and format for use as input for deep learning models.

[0073] S200, performing multi-feature extraction processing on the pre-processed patient laryngeal image based on a deep learning feature extraction model to obtain morphological features and quantitative features of the patient laryngeal image;

[0074] Specifically, raw laryngoscopic images were preprocessed and standardized to a uniform size and format before being input into a deep learning model for feature extraction. Based on in-depth literature research and expert experience, 14 features related to laryngeal diseases were extracted from laryngoscopic images. These features include eight deep learning-based features (texture, color, boundaries, vascular morphology, vascular color, vascular orientation, mucosal color, and lesion location) and six quantitative features (aspect ratio, color spectral information, S-channel image entropy, texture information, histogram of oriented gradients, and color moments). Texture, color, boundaries, vascular morphology, vascular color, vascular orientation, and mucosal color were acquired using a multi-task classifier constructed using a Transformer network with a cross-fusion encoder. Lesion location was extracted using the classic ResNet-50 deep residual network to identify laryngeal anatomical features and localize lesions.

[0075] It should be noted that, in some embodiments, step S200 may include: S210, performing feature extraction processing on the preprocessed patient laryngeal image using a Transformer network model based on a cross-fusion encoder to obtain morphological features of the first patient laryngeal image, where the morphological features of the first patient laryngeal image include texture features, color features, boundary features, vascular morphology features, vascular color features, vascular direction features, and mucosal color features;

[0076] Furthermore, it should be noted that, in some embodiments, step S210 may include: S211, inputting the preprocessed patient laryngeal image into a Transformer network model based on a cross-fusion encoder, the Transformer network model based on the cross-fusion encoder including several image feature extraction modules and linear layers, the several image feature extraction modules including a patch embedding layer and a cross-fusion encoder module; S212, performing dimensionality reduction processing on the preprocessed patient laryngeal image by the patch embedding layer of the Transformer network model based on the cross-fusion encoder to obtain a reduced-dimensional patient laryngeal image; S213, performing image feature extraction processing on the reduced-dimensional patient laryngeal image by the cross-fusion encoder module of the Transformer network model based on the cross-fusion encoder to obtain preliminary morphological features of the first patient laryngeal image;

[0077] Specifically, the reduced-dimensionality patient laryngeal image is input into the cross-fusion encoder module of the Transformer network model of the cross-fusion encoder, and the cross-fusion encoder module includes a first normalization layer, a parallel attention layer, a second normalization layer and a multi-layer perceptron; based on the first normalization layer and the second normalization layer of the cross-fusion encoder module, the reduced-dimensionality patient laryngeal image is subjected to feature extraction to obtain the patient laryngeal feature image; based on the parallel attention layer of the cross-fusion encoder module, the patient laryngeal feature image is divided into global feature information and fine-grained features and cross-spliced ​​to obtain the spliced ​​patient laryngeal image features; based on the multi-layer perceptron of the cross-fusion encoder module, the spliced ​​patient laryngeal image features are perceived and output to obtain preliminary morphological features of the first patient laryngeal image.

[0078] In this embodiment, if Figure 4 As shown in the figure, based on the Transformer model, a feature extraction network with a cross-fusion encoder module is designed. The network consists of 4 stages, each of which stacks a patch embedding layer (PatchEmbed) and an encoder module. Finally, a linear layer (LN) is used to predict the image category, which can be expressed as:

[0079]

[0080] In the above formula, Represents the input image, H, W, and C represent the width, height, and number of channels of the input image, respectively. In the first stage, the embedding layer first reduces the resolution of the input image to H1=H / N and W1=W / N. Based on empirical analysis, N=4 is set, and the number of channels is increased to C1. The encoder module performs attention calculation to extract image features. In each subsequent stage, non-overlapping 2×2 convolution kernels are used to perform space-to-depth operations to increase the dimension of the feature map and reduce the resolution. Then, a cross encoder module consisting of two sub-networks, parallel attention and multi-layer perceptron, is used for feature extraction. The cross fusion encoder uses two self-attention mechanisms placed in parallel for detail and global feature extraction, respectively. This layer is located between the two layer normalizations. The features of the input image are divided into two subsets, which process global information and fine-grained features respectively, which can be expressed as:

[0081] X i =Split(Z L ),i∈{0,1}

[0082] In the above formula, X i Represents the i-th feature subset obtained by cutting.

[0083] For the two feature subsets, we use inter-window attention to learn the global information of the image and self-attention within the window to learn fine-grained features. Then, we cross-join the self-attention outputs to ensure that each pixel in the final output feature map has both global and local information, which can be expressed as:

[0084] Z′ L+1 =Conv 1×1 {Contact[MHSA(X1),MHSA(X0)]}

[0085] Finally, the encoder output is obtained by forming a residual structure through LN and multi-layer perceptron (MLP), which can be expressed as:

[0086] Z L+1 =MLP(LN(Z′ L+1 ))+Z′ L+1

[0087] In the above formula, Z L+1 The morphological features of the output first patient laryngeal image are represented.

[0088] S214. Based on the linear layer of the Transformer network model of the cross-fusion encoder, perform image feature category prediction processing on the preliminary morphological features of the first patient's laryngeal image to obtain the morphological features of the first patient's laryngeal image.

[0089] S220, performing feature extraction processing on the preprocessed patient laryngeal image using a ResNet-50 deep residual network model to obtain morphological features of the second patient laryngeal image, where the morphological features of the second patient laryngeal image represent lesion location features;

[0090] S230, combining the morphological features of the first patient's laryngeal image with the morphological features of the second patient's laryngeal image to obtain the morphological features of the patient's laryngeal image;

[0091] S240. Perform lesion image analysis on the preprocessed patient laryngeal image to obtain quantitative features of the patient laryngeal image. The quantitative features of the patient laryngeal image include aspect ratio information, color spectrum information, S-channel image entropy information, texture information, directional gradient histogram information, and color moment information.

[0092] In some specific embodiments, in addition to the above-mentioned 8 deep learning-based features, 6 quantitative features are obtained by analyzing the lesion image, namely, aspect ratio: the ratio of width to height of the lesion area, reflecting the shape of the lesion; color spectrum information: the main color component extracted after converting the image color space; S channel image entropy: in the HSI color space, the image entropy value of the S channel reflects the color characteristics; texture information: the texture features of the lesion are extracted based on the local binary pattern (LBP) method; histogram of directional gradients: the lesion boundary and shape characteristics are captured through distribution information; color moment: reflecting the brightness, distribution area and symmetry of the color. Finally, 14 features related to the laryngeal image and the lesion are extracted for multi-scale feature fusion decision-making.

[0093] S300, performing multi-scale feature fusion processing on the morphological features and quantitative features of the patient's laryngeal image to obtain key features of the patient's laryngeal image with feature weights;

[0094] It should be noted that, in some embodiments, step S300 may include: S310, fusing the morphological features of the patient's laryngeal image with the quantitative features of the patient's laryngeal image to obtain multi-scale feature data of the patient's laryngeal image; S320, using a sampling method with replacement based on a random forest model, repeatedly and randomly selecting the multi-scale feature data of the patient's laryngeal image to construct a training sample subset; S330, training multiple decision trees based on the training sample subset, each tree randomly selecting features for splitting during the training process to construct a random forest model; S340, voting on the multi-scale feature data of the patient's laryngeal image based on the random forest model to obtain key features of the patient's laryngeal image with feature weights.

[0095] In some specific embodiments, Figure 5 As shown, the multi-scale feature fusion decision module uses a random forest model with sampling with replacement to generate multiple training sample subsets from image feature data. Each sample subset is used to train a decision tree. For each decision tree, at each node split, a subset of features is randomly selected from all features as candidate features for splitting. After multiple decision trees are independently trained, a random forest model is formed.

[0096] The relevant definitions are as follows:

[0097] Dataset: in Represents the feature vector of the sample, y i ∈{1,2,...,C} represents the category of the sample.

[0098] Decision tree model: T m (x) represents the classification result of the mth tree for sample x, m = 1, 2, ..., M, and M is the total number of decision trees.

[0099] Feature subset: At each splitting node, the randomly selected feature subset is recorded as where |F m |<<d.

[0100] By integrating multiple decision trees, the fusion decision of multi-scale features of laryngeal images can be achieved. The prediction result is determined by majority voting and can be expressed as:

[0101]

[0102] Among them, ∥(·) is the indicator function, T m When (x) = c, ∥ = 1, otherwise 0, Indicates the final classification result.

[0103] The final model will output a predicted category label, and by calculating the importance of each feature in the classification task, 6 key features are extracted to explain the evaluation basis of the model.

[0104] S400, performing human-computer interactive visualization processing on key features of the patient's larynx with feature weights to construct a patient's larynx recognition and diagnosis result;

[0105] It should be noted that, in some embodiments, step S400 may include: S410, outputting interpretable information based on the key features of the patient's larynx with feature weights, the interpretable information including texture information with weight ratio, tumor boundary information with weight ratio, vascular morphology information with weight ratio, S channel image entropy information with weight ratio, vascular color information with weight ratio, and vascular direction information with weight ratio; S420, performing visual significance labeling processing on the key features of the patient's larynx with feature weights based on the interpretable information to obtain the lesion area of ​​the patient's larynx; S430, performing confirmation feedback on the lesion area of ​​the patient's larynx to obtain the patient's larynx identification diagnosis result.

[0106] In some specific embodiments, Figure 6As shown in the figure, the human-computer interactive online learning is written in Python and consists of three parts: an interpretable disease diagnosis unit, a visual lesion display unit, and an online feedback optimization unit. The interpretable disease diagnosis unit can display the diagnosis results of the current case and the current image frame, and at the same time provide interpretable information, including six key features such as texture information, tumor boundary, vascular morphology, S-channel image entropy, vascular color, and vascular direction, as well as their proportions, to help physicians better understand the diagnosis results. In addition to being able to display the patient's laryngeal image in real time, the visual lesion display unit can also significantly annotate the lesion area based on deep learning features. The online feedback optimization unit allows physicians to provide feedback on the diagnosis results, annotate the diagnostic error features, dynamically update the training data set, and use the incremental learning algorithm to optimize the model parameters in real time to improve the model generalization ability.

[0107] In summary, the present invention utilizes a video laryngoscope system to standardize the collected patient laryngeal images. These images are then fed into a trained Transformer and ResNet-50 deep learning model. Based on prior knowledge, 14 features related to laryngeal tumors are extracted. A random forest model is then used to perform multi-scale feature fusion decision-making, outputting an auxiliary diagnosis result containing interpretable information. Finally, a graphically interactive platform is developed to display diagnostic results and key features in real time. An incremental learning mechanism is also introduced to dynamically update the model dataset based on physician feedback, improving model accuracy.

[0108] The advantages of the embodiments of the present invention compared to the prior art are:

[0109] 1) Improve diagnostic accuracy. The artificial intelligence model used is based on a lesion feature recognition system that can automatically identify and classify lesion features. By making secondary judgments on the combination of identified local lesion features, it can guide clinical decision-making, obtain more accurate lesion diagnosis results, reduce misdiagnosis rates, and improve diagnostic performance.

[0110] 2) Increased diagnostic transparency. Previous AI endoscopic diagnostic systems could only return unexplained final decision results, making it difficult for endoscopists to learn from the models. Furthermore, the poor interpretability of black-box models severely limited the clinical application of AI systems. This embodiment of the present invention utilizes an explainable AI system to diagnose early-stage laryngeal cancer. By using feature extraction to concretize abstract diagnostic theories, it provides endoscopists with diagnostic results and evidence, thereby increasing diagnostic transparency.

[0111] See also Figure 2 The embodiment of the present application further provides a laryngoscope image recognition system based on multi-feature extraction, which can implement the above-mentioned laryngoscope image recognition method based on multi-feature extraction. The system includes:

[0112] The first module 201 is used to obtain a patient's laryngeal image and perform image data preprocessing to obtain a preprocessed patient's laryngeal image;

[0113] The second module 202 is configured to perform multi-feature extraction processing on the pre-processed patient laryngeal image based on a deep learning feature extraction model to obtain morphological features and quantitative features of the patient laryngeal image;

[0114] The third module 203 is configured to perform multi-scale feature fusion processing on the morphological features of the patient's laryngeal image and the quantitative features of the patient's laryngeal image to obtain key features of the patient's laryngeal image with feature weights;

[0115] The fourth module 204 is used to perform human-computer interactive visualization processing on the key features of the patient's larynx with feature weights to construct the patient's larynx identification and diagnosis results.

[0116] It can be understood that the contents of the above method embodiments are all applicable to the present system embodiments, the functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0117] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but are not intended to limit the scope of the present invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and essence of the present invention should be within the scope of the present invention.

Claims

1. A laryngoscope image recognition method based on multi-feature extraction, characterized in that: The method comprises the following steps: Acquiring a laryngeal image of the patient and performing image data preprocessing to obtain a preprocessed laryngeal image of the patient; Performing multi-feature extraction processing on the pre-processed patient laryngeal image based on a deep learning feature extraction model to obtain morphological features and quantitative features of the patient laryngeal image; performing multi-scale feature fusion processing on the morphological features of the patient's laryngeal image and the quantitative features of the patient's laryngeal image to obtain key features of the patient's laryngeal image with feature weights; Performing human-computer interactive visualization processing on the key features of the patient's larynx with the feature weights to construct a patient's larynx identification and diagnosis result; The multi-feature extraction process based on the deep learning feature extraction model is performed on the pre-processed patient laryngeal image to obtain the morphological features and quantitative features of the patient laryngeal image, including: Performing feature extraction processing on the preprocessed patient laryngeal image through a Transformer network model based on a cross-fusion encoder to obtain morphological features of the first patient laryngeal image, wherein the morphological features of the first patient laryngeal image include texture features, color features, boundary features, vascular morphology features, vascular color features, vascular direction features, and mucosal color features; the Transformer network model based on the cross-fusion encoder includes several image feature extraction modules and linear layers, and several of the image feature extraction modules include patch embedding layers and cross-fusion encoder modules; the cross-fusion encoder module includes a first normalization layer, a parallel attention layer, a second normalization layer, and a multilayer perceptron; performing feature extraction processing on the preprocessed patient laryngeal image using a ResNet-50 deep residual network model to obtain morphological features of a second patient laryngeal image, wherein the morphological features of the second patient laryngeal image represent lesion location features; combining the morphological features of the first patient's laryngeal image and the morphological features of the second patient's laryngeal image to obtain the morphological features of the patient's laryngeal image; The preprocessed patient laryngeal image is subjected to lesion image analysis to obtain quantitative features of the patient laryngeal image, wherein the quantitative features of the patient laryngeal image include aspect ratio information, color spectrum information, S-channel image entropy information, texture information, directional gradient histogram information, and color moment information.

2. The method according to claim 1, characterized in that The step of acquiring a laryngeal image of a patient and performing image data preprocessing to obtain a preprocessed laryngeal image of the patient includes: Obtain images of the patient's larynx using an electronic laryngoscope device; Extracting clear key frames of the patient's laryngeal image and filtering out blurry frames and non-target areas of the patient's laryngeal image to construct a coarsely screened patient's laryngeal image; Performing accuracy testing on the coarsely screened patient laryngeal image to obtain a standard patient laryngeal image; The standard patient laryngeal image is subjected to image enhancement and formatting processing to obtain the pre-processed patient laryngeal image.

3. The method according to claim 2, characterized in that The performing accuracy testing on the coarsely screened patient laryngeal image to obtain a standard patient laryngeal image includes: An image clarity evaluation function is constructed based on the Laplace operator, and image blur detection processing is performed on the coarsely screened patient laryngeal image to obtain a clear patient laryngeal image; Calculating the ratio of valid matching points of the clear patient laryngeal image based on the ORB algorithm to obtain an image repeatability score; performing an image repeatability test on the clear patient laryngeal image based on a preset scoring threshold and in combination with the image repeatability score to obtain a filtered patient laryngeal image; The filtered patient laryngeal image is subjected to laryngeal key structural feature detection by a target detection algorithm to obtain the standard patient laryngeal image.

4. The method according to claim 1, wherein The step of performing feature extraction processing on the preprocessed patient laryngeal image by using a Transformer network model based on a cross-fusion encoder to obtain morphological features of the first patient laryngeal image includes: Inputting the preprocessed patient laryngeal image into the Transformer network model based on the cross-fusion encoder; performing dimensionality reduction processing on the preprocessed patient laryngeal image based on a patch embedding layer of a Transformer network model of the cross-fusion encoder to obtain a dimensionality-reduced patient laryngeal image; performing image feature extraction processing on the dimensionality-reduced patient laryngeal image based on a cross-fusion encoder module of a Transformer network model of the cross-fusion encoder to obtain preliminary morphological features of the first patient laryngeal image; Based on the linear layer of the Transformer network model of the cross-fusion encoder, image feature category prediction processing is performed on the morphological features of the preliminary first patient laryngeal image to obtain the morphological features of the first patient laryngeal image.

5. The method according to claim 4, characterized in that The cross-fusion encoder module of the Transformer network model based on the cross-fusion encoder performs image feature extraction processing on the dimensionality-reduced patient laryngeal image to obtain preliminary morphological features of the first patient laryngeal image, including: Inputting the dimensionally reduced patient laryngeal image into the cross-fusion encoder module of the Transformer network model of the cross-fusion encoder; Based on the first normalization layer and the second normalization layer of the cross-fusion encoder module, feature extraction is performed on the dimensionally reduced patient laryngeal image to obtain a patient laryngeal feature image; Based on the parallel attention layer of the cross-fusion encoder module, the patient's laryngeal feature image is divided into global feature information and fine-grained features and cross-splicing is performed to obtain a spliced ​​patient laryngeal image feature; Based on the multi-layer perceptron of the cross-fusion encoder module, the features of the spliced ​​patient laryngeal image are perceived and output to obtain the morphological features of the preliminary first patient laryngeal image.

6. The method according to claim 1, characterized in that The multi-scale feature fusion processing is performed on the morphological features of the patient's laryngeal image and the quantitative features of the patient's laryngeal image to obtain key features of the patient's laryngeal image with feature weights, including: fusing the morphological features of the patient's laryngeal image with the quantitative features of the patient's laryngeal image to obtain multi-scale feature data of the patient's laryngeal image; Based on the random forest model, a sampling method with replacement was used to repeatedly randomly select multi-scale feature data of the patient's laryngeal images to construct a training sample subset. Training multiple decision trees based on the training sample subset, each tree randomly selecting features to split during the training process, and constructing a random forest model; Voting and decision-making are performed on the multi-scale feature data of the patient's laryngeal image based on the random forest model to obtain the key features of the patient's laryngeal image with feature weights.

7. The method according to claim 6, characterized in that The voting decision expression of the random forest model is specifically as follows: ； In the above formula, represents the final classification result, Indicates the Tree pairs sample The classification results, represents the number of sample categories, represents the sample category, Represents the indicator function.

8. The method according to claim 1, characterized in that The performing human-computer interactive visualization processing on the key features of the patient's larynx with the feature weights to construct the patient's larynx identification and diagnosis results includes: Outputting interpretable information based on the key features of the patient's larynx with the feature weights, the interpretable information including texture information with weighted proportions, tumor boundary information with weighted proportions, vascular morphology information with weighted proportions, S-channel image entropy information with weighted proportions, vascular color information with weighted proportions, and vascular direction information with weighted proportions; Performing visual significance labeling processing on the key features of the patient's larynx with the feature weights according to the interpretable information to obtain the lesion area of ​​the patient's larynx; Confirmation feedback is performed on the lesion area of ​​the patient's throat to obtain the patient's throat identification diagnosis result.

9. A laryngoscope image recognition system based on multi-feature extraction, characterized in that: The system comprises: The first module is used to obtain a laryngeal image of a patient and perform image data preprocessing to obtain a preprocessed laryngeal image of the patient; The second module is used to perform multi-feature extraction processing on the pre-processed patient laryngeal image based on a deep learning feature extraction model to obtain morphological features and quantitative features of the patient laryngeal image; A third module is configured to perform multi-scale feature fusion processing on the morphological features of the patient's laryngeal image and the quantitative features of the patient's laryngeal image to obtain key features of the patient's laryngeal image with feature weights; The fourth module is used to perform human-computer interactive visualization processing on the key features of the patient's larynx with the feature weights to construct a patient's larynx identification and diagnosis result; The multi-feature extraction process based on the deep learning feature extraction model is performed on the pre-processed patient laryngeal image to obtain the morphological features and quantitative features of the patient laryngeal image, including: Performing feature extraction processing on the preprocessed patient laryngeal image through a Transformer network model based on a cross-fusion encoder to obtain morphological features of the first patient laryngeal image, wherein the morphological features of the first patient laryngeal image include texture features, color features, boundary features, vascular morphology features, vascular color features, vascular direction features, and mucosal color features; the Transformer network model based on the cross-fusion encoder includes several image feature extraction modules and linear layers, and several of the image feature extraction modules include patch embedding layers and cross-fusion encoder modules; the cross-fusion encoder module includes a first normalization layer, a parallel attention layer, a second normalization layer, and a multilayer perceptron; performing feature extraction processing on the preprocessed patient laryngeal image using a ResNet-50 deep residual network model to obtain morphological features of a second patient laryngeal image, wherein the morphological features of the second patient laryngeal image represent lesion location features; combining the morphological features of the first patient's laryngeal image and the morphological features of the second patient's laryngeal image to obtain the morphological features of the patient's laryngeal image; The preprocessed patient laryngeal image is subjected to lesion image analysis to obtain quantitative features of the patient laryngeal image, wherein the quantitative features of the patient laryngeal image include aspect ratio information, color spectrum information, S-channel image entropy information, texture information, directional gradient histogram information, and color moment information.

Images