Multi-mode ear-nose-throat patient rehabilitation scheme optimization method and system

Abstract

The invention provides a multi-mode ear-nose-throat patient rehabilitation scheme optimization method and system, and the method comprises the steps: S1, obtaining the ear-nose-throat pathology data of a patient, carrying out the preprocessing operation of the ear-nose-throat pathology data, carrying out the fusion, generating a rehabilitation feature tensor, comprehensively integrating the physiological signals, image features and clinical text data of the patient, and carrying out the recognition of the rehabilitation feature tensor; the limitation of a single data type is broken through, iterative optimization is carried out through the rehabilitation prediction model constructed in the steps S2 and S3 in combination with an improved genetic algorithm, the rehabilitation effect is accurately predicted, training parameters are dynamically adjusted, the accuracy and effectiveness of rehabilitation treatment are improved, finally, quantitative evaluation indexes and evaluation are combined through the step S4, and the rehabilitation treatment accuracy and effectiveness are improved. And the scientificity of the scheme is judged according to a mean threshold value, so that the fitness, implementation safety and expected effect reasonability of the scheme are guaranteed, the treatment risk is reduced, and a safer and more efficient auxiliary rehabilitation scheme is provided for the patient.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, in particular to a multi-modal ear-nose-throat patient rehabilitation scheme optimization method and system. BACKGROUND

[0002] With the increasing demand for personalized rehabilitation of ear-nose-throat disease patients and the increasing complexity of rehabilitation treatment, precise and personalized rehabilitation scheme development has become a core technical challenge in the modern ear-nose-throat rehabilitation field. Traditional rehabilitation schemes often have limitations in the development process, and fixed training strategies that rely on experience cannot adapt to the differences in individual physiological characteristics, anatomical structures, and medical history of patients, resulting in deviations between rehabilitation schemes and actual patient needs, affecting rehabilitation effectiveness.

[0003] Therefore, it is urgent to develop an ear-nose-throat pathological data fusion mechanism and an intelligent optimization model to build a rehabilitation scheme optimization framework that can integrate physiological, imaging, and text data and has dynamic adjustment capabilities to achieve efficient, safe, and personalized rehabilitation treatment. SUMMARY

[0004] In view of the above-mentioned problems, in combination with the first aspect of the present application, the embodiments of the present application provide a multi-modal ear-nose-throat patient rehabilitation scheme optimization method, which comprises: S1: obtaining ear-nose-throat pathological data of a patient, and performing preprocessing operations on the ear-nose-throat pathological data, performing feature fusion on the preprocessed ear-nose-throat pathological data, and obtaining a rehabilitation feature tensor; S2: analyzing the rehabilitation feature tensor based on a rehabilitation decision mechanism to obtain an initial rehabilitation scheme; S3: constructing a rehabilitation prediction model based on a rehabilitation database, inputting the initial rehabilitation scheme into the rehabilitation prediction model and optimizing it to obtain an optimized rehabilitation scheme; S4: performing professional evaluation on the optimized rehabilitation scheme, and executing a rehabilitation feedback mechanism based on the evaluation results.

[0005] As a further scheme of the present application, the ear-nose-throat pathological data of a patient is obtained, and preprocessing operations are performed on the ear-nose-throat pathological data, feature fusion is performed on the preprocessed ear-nose-throat pathological data, and a rehabilitation feature tensor is obtained, comprising: The ear-nose-throat pathological data includes physiological signal data, imaging data, and clinical text data of the patient; The physiological signal data, imaging data, and clinical text data of the patient are subjected to data preprocessing operations respectively to obtain physiological feature sequences, imaging feature matrices, and text feature vectors; A feature fusion model is constructed, and the physiological feature sequences, imaging feature matrices, and text feature vectors are input into the feature fusion model to obtain the rehabilitation feature tensor.

[0006] As a further scheme of the present application, the physiological signal data, the imaging data and the clinical text data of the patient are respectively subjected to data preprocessing operations to obtain physiological feature sequences, imaging feature matrices and text feature vectors, including: Baseline drift and high-frequency noise in the physiological signal data are removed based on wavelet transform, while motion artifacts and power frequency interference are eliminated based on adaptive filtering operation to obtain preliminary physiological signals, the preliminary physiological signals are subjected to signal segmentation based on resolution, and feature extraction is performed on each segmented signal to obtain physiological feature sequences; The region of interest in the imaging data is extracted based on a U-net model, and feature point matching is performed on the region of interest based on a scale-invariant feature transform algorithm to generate an imaging feature matrix; The clinical text data is subjected to word segmentation and part-of-speech tagging based on natural language processing, and semantic encoding is performed on the clinical text data based on a bidirectional long short-term memory network to obtain a text feature vector; The physiological feature sequences, the imaging feature matrices and the text feature vector are subjected to multiple alignment based on multi-dimensional collaborative processing; The multi-dimensional collaborative processing includes establishing a time synchronization relationship between the physiological feature sequences and the imaging feature matrices based on a dynamic time warping algorithm, spatially registering three-dimensional coordinates of all imaging feature matrices based on a set projection model, and constructing a semantic association relationship between the text feature vector and the imaging feature matrix.

[0007] As a further scheme of the present application, a feature fusion model is constructed, including: An ear-nose-throat interaction mechanism is composed of time-space interaction, time-language interaction and space-language interaction, and a feature fusion model is constructed based on the ear-nose-throat interaction mechanism; The time-space interaction is represented by projecting the physiological feature sequences into the imaging feature matrices based on variable convolution, the time-language interaction is represented by labeling the text features in the physiological feature sequences based on cross-modal attention, and the space-language interaction is represented by adjusting the association strength between the imaging feature matrix and the text feature dimension based on a gating mechanism.

[0008] As a further scheme of the present application, a rehabilitation feature tensor is analyzed based on a rehabilitation decision mechanism to obtain an initial rehabilitation scheme, including: The rehabilitation feature tensor is mapped into a multi-dimensional state feature vector, and the multi-dimensional state feature vector is input into a state space to generate a decision scheme; The state space contains 120 action spaces, and the 120 action spaces are obtained by Cartesian product operation based on five rehabilitation training intensities, three training frequencies and eight types of training actions; acquire historical patient records of the same kind of cases as the current patient, acquire the average rate of rehabilitation effect improvement of the historical patient discipline corresponding to the patient using the current decision scheme, the matching degree of the current decision scheme and the current patient, and the safety score of the current scheme, and construct a reward function, output the safety score of the current patient using the current decision scheme; extract the three decision schemes with the highest safety scores in all decision schemes, and form an initial rehabilitation scheme.

[0009] As a further scheme of the present application, a rehabilitation prediction model is constructed based on the rehabilitation database, and the initial rehabilitation scheme is input into the rehabilitation prediction model and optimized to obtain an optimized rehabilitation scheme, comprising: The rehabilitation prediction model is constructed based on gradient boosting decision trees and recurrent neural networks; The gradient boosting decision trees perform feature importance sorting through regression trees, and filter out key features with a weight of the top 30% in the rehabilitation effect, and the recurrent neural networks obtain the time sequence dependence of the rehabilitation state through the gating mechanism; Meanwhile, the initial rehabilitation scheme is input into the rehabilitation prediction model to obtain a predicted rehabilitation state curve; The initial rehabilitation scheme is optimized based on the predicted rehabilitation state curve using an improved genetic algorithm to obtain an optimized rehabilitation scheme.

[0010] As a further scheme of the present application, the initial rehabilitation scheme is optimized based on the predicted rehabilitation state curve using an improved genetic algorithm to obtain an optimized rehabilitation scheme, comprising: The training parameters in the initial rehabilitation scheme are encoded using real number coding chromosomes based on the improved genetic algorithm, and three objective functions of rehabilitation period, training pain and functional recovery are set, and patient adaptation constraints are added; The initial rehabilitation scheme is iteratively optimized based on adaptive crossover probability and elite reservation strategy to obtain an optimized rehabilitation scheme.

[0011] As a further scheme of the present application, the optimized rehabilitation scheme is evaluated professionally, and a rehabilitation feedback mechanism is executed based on the evaluation results, comprising: Ear-nose-throat quantitative evaluation indexes of scheme adaptation degree, implementation safety and rationality of expected effect are constructed, the optimized rehabilitation scheme is scored based on the ear-nose-throat quantitative evaluation indexes, all sub-index scores of the ear-nose-throat quantitative evaluation indexes are acquired, the mean value of the sub-index scores is calculated, the mean value threshold and the sub-index score threshold are set, and the mean value and the sub-index score are compared with the corresponding threshold respectively: When the mean value is greater than the mean value threshold and all sub-index scores are greater than the sub-index score threshold, it is determined that the evaluation is passed, and the patient is treated with the aid of the optimized rehabilitation scheme; When any one of the mean value and all sub-index scores is less than or equal to the corresponding threshold value, it is determined that the evaluation is failed, external feedback is obtained, the external feedback is converted into a quantitative constraint condition, and the value step S2 is returned.

[0012] In still another aspect, the embodiment of the present application also provides a multi-modal ear-nose-throat patient rehabilitation scheme optimization system, comprising: An acquisition module is configured to acquire ear-nose-throat pathological data of a patient, input an initial rehabilitation scheme into a rehabilitation prediction model and perform optimization, and acquire an optimized rehabilitation scheme; A processing module is configured to pre-process the ear-nose-throat pathological data; A fusion module is configured to perform feature fusion on the pre-processed ear-nose-throat pathological data and acquire a rehabilitation feature tensor; A construction module is configured to construct a rehabilitation prediction model based on a rehabilitation database; An analysis module is configured to analyze the rehabilitation feature tensor based on a rehabilitation decision mechanism and acquire an initial rehabilitation scheme; An evaluation module is configured to perform professional evaluation on the optimized rehabilitation scheme and execute a rehabilitation feedback mechanism based on an evaluation result.

[0013] Based on the above aspects, the embodiment of the present application realizes ear-nose-throat pathological data acquisition and fusion, fully integrates physiological signals, image features and clinical text data of a patient, breaks the limitation of a single data type, makes the generated scheme more in line with individual characteristics of the patient, and avoids deviation of the scheme caused by individual differences; a rehabilitation prediction model constructed based on a gradient boosting decision tree and a recurrent neural network is iteratively optimized by combining an improved genetic algorithm, rehabilitation effects are accurately predicted and training parameters are dynamically adjusted, an optimal balance between a rehabilitation period, training pain and functional recovery of the scheme is ensured, and the accuracy and effectiveness of rehabilitation treatment are improved; quantitative evaluation indexes are combined with doctor evaluation, the scientificity of the scheme is judged according to a mean value threshold, the role of clinical experience of doctors is played, the scheme is continuously optimized through feedback iteration of quantitative constraint conditions, and the adaptation degree, implementation safety and rationality of expected effects of the scheme are ensured, treatment risks are reduced, and a safer and more efficient auxiliary rehabilitation scheme is provided for the patient. BRIEF DESCRIPTION OF DRAWINGS

[0014] Figure 1 is an execution flow schematic diagram of a multi-modal ear-nose-throat patient rehabilitation scheme optimization method provided by the embodiment of the present application.

[0015] Figure 2 is a principle block diagram of step S4 in a multi-modal ear-nose-throat patient rehabilitation scheme optimization method provided by the embodiment of the present application.

[0016] Figure 3is a schematic diagram of a multi-modal ear-nose-throat patient rehabilitation scheme optimization system provided by an embodiment of the present application. DETAILED DESCRIPTION

[0017] The present application will be described in detail below with reference to the accompanying drawings, Figure 1 is an execution flow schematic diagram of a multi-modal ear-nose-throat patient rehabilitation scheme optimization method provided by an embodiment of the present application, Figure 2 is a principle block diagram of step S4 in a multi-modal ear-nose-throat patient rehabilitation scheme optimization method provided by an embodiment of the present application, and the multi-modal ear-nose-throat patient rehabilitation scheme optimization method will be described in detail below.

[0018] Specifically, a multi-modal ear-nose-throat patient rehabilitation scheme optimization method comprises: Step S1, ear-nose-throat pathological data of a patient is acquired, and pre-processing operation is performed on the ear-nose-throat pathological data, the ear-nose-throat pathological data after pre-processing is subjected to feature fusion, and a rehabilitation feature tensor is acquired.

[0019] In this embodiment, step S1 comprises: Step S11, the ear-nose-throat pathological data comprises physiological signal data, imaging data and clinical text data of the patient.

[0020] Specifically, the auditory potential signal is collected by the acquisition module, and the sampling frequency can be set to 2000 Hz, and the collected auditory potential signal is used to reflect the functional state of the auditory pathway; the nasal airflow signal is collected by the acquisition module, and the collection range is-500 Pa to 500 Pa, and the acquired nasal airflow signal is used to record the airflow pressure change when the nasal cavity is ventilated; the laryngeal electromyography signal is acquired according to the acquisition module, and the acquired laryngeal electromyography signal is used to reflect the electrophysiological change when the laryngeal muscle is active, and the above-mentioned auditory potential signal, nasal airflow signal and laryngeal electromyography signal constitute the physiological signal data, which can directly reflect the functional state of the patient's ear-nose-throat and can be used to evaluate the rehabilitation effect. The high-resolution CT image is acquired by the acquisition module, and the high-resolution CT image can reflect the anatomical structure details of the ear-nose-throat; the endoscope video is shot according to the acquisition module, and the endoscope video frame is acquired, and the resolution is 1920*1080 pixels, and the frame rate is 30 frames per second, and the surface condition of the ear-nose-throat mucosa and other tissues can be observed through the endoscope video frame, and the high-resolution CT image and the endoscope video frame constitute the imaging data, which can provide anatomical structure information and help to develop the rehabilitation scheme. The patient's medical history record, medication list and rehabilitation training log are obtained through the hospital information system extraction or manual input of medical staff into the text input terminal, and the patient's medical history record, medication list and rehabilitation training log constitute the clinical text data, which provide background support for subsequent rehabilitation scheme development.

[0021] In some possible embodiments, assuming that ear-nose-throat pathological data collection is performed on patient A, including: patient A has a latency period of 1.2 milliseconds longer than a normal reference value and a wave amplitude reduction under the stimulation of a 2000-hertz sound; nasal cavity airflow dynamics signals show that the nasal cavity pressure fluctuation is between -100 Pa and 80 Pa when patient A is breathing calmly, the maximum negative pressure is 350 Pa when patient A inhales deeply, and the ventilation symmetry is good; the laryngeal electromyography signals show that the electromyography amplitude of the laryngeal muscles on both sides is 0.3 mV when patient A pronounces "ah", and the electromyography peak value reaches 0.8 mV when patient A swallows. The high-resolution CT image shows that the mucosa thickness of the pharynx and larynx of patient A is 3.5 mm, which is less than 2 mm, and multiple lymphoid follicle hyperplasia shadows with a diameter of 2-3 mm are visible on the posterior wall of the pharynx; in the endoscopic video frames, the 15th second frame clearly shows that the bilateral vocal cords of the larynx are slightly congested, the edges are slightly thickened, and there is a 0.5-mm gap when the glottis is closed. The medical history record shows that patient A has a 5-year history of chronic pharyngitis, 2-3 acute episodes per year, no drug allergy history, and a 10-year smoking history; the medication list records that patient A has taken xx capsules and xx pills in the past three months; the rehabilitation training log shows that patient A received nebulization treatment one month ago, and also performed simple voice training, such as reading for 10 minutes per day.

[0022] In step S12, the physiological signal data, the imaging data, and the clinical text data of the patient are respectively preprocessed to obtain a physiological feature sequence, an imaging feature matrix, and a text feature vector.

[0023] In this embodiment, step S12 includes: In step S121, baseline drift and high-frequency noise in the physiological signal data are removed based on wavelet transform, motion artifacts and power frequency interference are eliminated based on adaptive filtering operation, a preliminary physiological signal is obtained, the preliminary physiological signal is segmented based on resolution, and features of each segmented signal are extracted to obtain a physiological feature sequence.

[0024] Specifically, the auditory potential signal, the nasal airflow signal, and the laryngeal electromyography signal are processed by wavelet transform, baseline drift and high-frequency noise in the signals are removed through decomposition and reconstruction, and effective physiological signal components are retained at the same time; adaptive filtering technology is then used on the physiological signal data after wavelet transform, filtering parameters are dynamically adjusted according to the frequency characteristics of the power frequency interference, and the power frequency interference of 50 Hz or 60 Hz is eliminated; then the denoised signal is segmented, and the signal is divided into several segments according to the period of the physiological signal or the characteristic change point, and the feature points of each segment, such as the peak value, the valley value, and the rising slope, are extracted, and the standardized physiological feature sequence is output.

[0025] In some possible embodiments, assuming that there is baseline drift and high-frequency noise in the auditory potential raw signal of the ear-nose-throat patient A, after wavelet transform processing, the signal smoothness is improved and the baseline tends to be stable; after eliminating power frequency interference through adaptive filtering, the interference component of 50Hz basically disappears; after segmenting and extracting feature points from the processed signal, the obtained standardized physiological feature sequence has characteristic parameters such as a peak value appearance time of 15ms and a peak value amplitude of 5μV at 2000Hz stimulation; after pre-processing of the nasal airflow signal, information such as airflow pressure peak value 80Pa at quiet breathing, pressure valley value-350Pa at inhalation and pressure change rate is obtained; the obtained laryngeal electromyographic signal includes characteristics such as 0.3mV during sound production, 0.8mV during swallowing and duration.

[0026] In step S122, the region of interest in the imaging data is extracted based on the U-net model, and feature point matching is performed on the region of interest based on the scale-invariant feature transform algorithm to generate an image feature matrix.

[0027] Specifically, through the image segmentation model of the U-Net, the regions of interest of the ear-nose-throat, such as the pharyngeal mucosa, the auditory ossicles and the nasal cavity channel, are identified and extracted through the coding-decoding structure and the skip connection; the feature point detection and description of the region of interest are performed in different scale spaces through the scale-invariant feature transform algorithm, the influence of changes such as image scale and rotation is eliminated through feature point matching, and the image feature matrix is generated.

[0028] It should be noted that the input of the U-Net model is a pre-processed image, the left side of the model is a series of convolution and pooling operations constituting a contraction path for extracting features, each convolution block is followed by a maximum pooling operation, with the increase of network depth, the size of the feature map decreases, and the number of channels increases, at the bottom of the contraction path, there is a feature extraction layer, followed by an expansion path which contains a series of up-sampling and convolution operations for reconstructing the details and size of the image, in the expansion path, the feature map gradually increases through up-sampling, and the number of channels decreases. After each step of up-sampling, the corresponding feature map in the contraction path is spliced, which helps the network to learn accurate location information, and finally the model outputs a feature map with the same size as the input image, representing the class of each pixel. The scale-invariant feature transform is an algorithm for image processing, which can find feature points in different scale spaces and match them.

[0029] In some possible embodiments, it is assumed that the U-Net model is used to extract the pharyngeal mucosa thickening area and the pharyngeal wall lymphoid follicle hyperplasia area and other regions of interest from the high-resolution CT image of patient A; the regions of interest are processed by the scale-invariant feature transformation algorithm, and an image feature matrix element is generated, the image feature matrix element includes, for example, a texture feature with a gray distribution variance of 20 of the mucosa thickening area, a shape feature with a circularity of 0.8 of the lymphoid follicle, and the like; similarly, after the endoscopic video frame is preprocessed, the extracted region of interest such as the bilateral vocal cords of the throat includes, for example, an area of 0.2 square centimeters of the congested vocal cord, an edge contour feature, and the like.

[0030] In step S123, the clinical text data is segmented and part-of-speech tagged based on natural language processing, and the clinical text data is semantically encoded based on a bidirectional long short-term memory network to obtain a text feature vector.

[0031] Specifically, the clinical text data is segmented by natural language processing to split continuous text into meaningful words or phrases, and part-of-speech tagging is performed, such as nouns, verbs, adjectives, etc.; the segmented text is semantically encoded by a bidirectional long short-term memory network combined with the semantic information of the preceding and following texts to convert the text information into a fixed-dimension structured text feature vector, which can represent the semantic content and context association of the text.

[0032] In some possible embodiments, it is assumed that after the medical history record of patient A “chronic pharyngitis history of 5 years, acute attack 2-3 times per year, no drug allergy history, and 10-year smoking history” is segmented and part-of-speech tagged, the result “chronic pharyngitis-noun, history-noun, 5 years-quantifier...” is obtained; after semantic encoding by the bidirectional long short-term memory network, it is converted into a 128-dimensional structured text feature vector, and the text feature vector has a higher feature value in the dimension corresponding to the key information “chronic pharyngitis” and “smoking history”.

[0033] In step S124, the physiological feature sequence, the image feature matrix, and the text feature vector are multi-aligned based on multi-dimensional collaborative processing.

[0034] The multi-dimensional collaborative processing includes establishing a time synchronization relationship between the physiological feature sequence and the image feature matrix based on a dynamic time warping algorithm, spatially registering three-dimensional coordinates of all image features based on a set projection model, and constructing a semantic association matrix of text features and image features.

[0035] Specifically, the sampling frequency difference between the physiological feature sequence and the image feature matrix is eliminated by calculating the distance matrix of the two types of feature sequences, finding the optimal path through time synchronization of the dynamic time warping algorithm, and nonlinearly aligning the time according to the example matrix and the optimal path. For example, the peak time of the electromyography signal is forced to match the key frame of the vocal cord closure in the video frame, eliminating the time offset and keeping the physiological function changes and the image dynamic features synchronized in the time dimension. Through spatial registration of the set projection model, the coordinates of different modalities in the unified image feature matrix are unified. By selecting anatomical landmark points as the reference, a spatial transformation matrix is established to project the three-dimensional features of the CT and the two-dimensional features of the endoscope to the standard coordinate system. For example, the three-dimensional coordinates (x1, y1, z1) of the lymphoid follicle on the posterior pharyngeal wall in the CT are associated with the two-dimensional coordinates (x2, y2) of the corresponding area in the endoscope through projection transformation, realizing accurate correspondence of the spatial position. When constructing the semantic association matrix of the text features and the image features, the anatomical parts and pathological state keywords are extracted from the text feature vector through natural language processing, and the similarity between the feature vectors of the corresponding areas in the image feature matrix is calculated. According to the similarity, the semantic association matrix is constructed to clearly match the relationship between the text description and the image features.

[0036] In some possible embodiments, taking patient A as an example, the sampling frequency of the laryngeal electromyography signal of patient A is 1000 Hz, a feature sequence containing 2000 time points is generated, the frame rate of the endoscope video frame is 30 fps, corresponding to an image feature matrix of 600 time points, the Euclidean distance matrix of the two is calculated through the dynamic time warping algorithm, it is found that the electromyography signal appears a swallowing action peak at 500 ms, corresponding to the complete closure state of the vocal cord in the 15th frame of the video frame, the local sequence of the electromyography signal is stretched to align the peak time with the 15th frame, so that the time error is controlled within 50 ms. In the spatial registration, the external auditory canal opening and the vertex of the nasal septum are selected as the reference points, and their CT coordinates are (50, 30, 20) and (45, 50, 15) respectively. The spatial transformation matrix of the CT and the endoscope is established. The three-dimensional coordinates (38, 42, 18) of the lymphoid follicle on the posterior pharyngeal wall in the CT are matched with the two-dimensional coordinates (220, 310) of the same follicle in the endoscope video after projection transformation, and the spatial error is within 2 mm, ensuring that the image feature of “lymphoid follicle hyperplasia” points to the same anatomical position in space. When constructing the semantic association matrix, the keyword vector is extracted from the clinical text “vocal cord mild congestion, lymphoid follicle hyperplasia on the posterior pharyngeal wall”, and the similarity with the image feature matrix is calculated. The similarity between the text feature of “vocal cord congestion” and the RGB feature vector of the vocal cord area in the endoscope image is 0.82, and the similarity between the text feature of “lymphoid follicle hyperplasia” and the shape feature vector of the follicle area in the CT image is 0.76. The 3x3 association matrix shows the semantic matching relationship.

[0037] Step S13, constructing a feature fusion model, inputting the physiological feature sequence, the image feature matrix and the text feature vector into the feature fusion model to obtain a rehabilitation feature tensor.

[0038] In this embodiment, step S13 includes: Step S131, based on the ear-nose-throat interaction mechanism composed of space-time interaction, time-language interaction and space-language interaction, constructing a feature fusion model based on the ear-nose-throat interaction mechanism; Wherein, the space-time interaction is represented as projecting the physiological feature sequence into the image feature matrix based on variable convolution, the time-language interaction is represented as labeling the text feature in the physiological feature sequence based on cross-modal attention, and the space-language interaction is represented as adjusting the correlation strength of the image feature matrix and the text feature dimension based on the gating mechanism.

[0039] Specifically, in the space-time interaction, the physiological feature sequence can be projected into the image feature matrix through variable convolution. The variable convolution adjusts the convolution kernel shape dynamically to adapt to the morphological difference between the physiological signal and the image feature, so that the physiological feature and the image feature form a correspondence in the spatial dimension. The time-language interaction labels the text feature on the key time point of the physiological feature sequence through cross-modal attention, and calculates the attention weight of the text and the physiological feature at the same time, highlighting the time correlation between the text description and the physiological change. The space-language interaction adjusts the correlation strength of the image feature matrix and the text feature dimension by using the gating mechanism. The gating unit dynamically outputs the weight according to the matching degree of the text keyword and the image region feature, strengthens the role of the related image feature and suppresses the irrelevant information.

[0040] In some possible embodiments, also taking patient A as an example, patient A has chronic pharyngitis and mild hearing loss, in the space-time interaction, the patient's laryngeal electromyography signal sequence is projected into the endoscope image feature matrix, so that the intensity change of the electromyography signal and the image region brightness change of the vocal cord movement form a spatial correspondence; in the time-language interaction, the cross-modal attention calculates the attention weight of the text "throat discomfort when swallowing" and the electromyography signal at the 500ms peak value time point as 0.8, and labels the text feature at the corresponding time point; the space-language interaction adjusts the text "lymphoid follicle hyperplasia in the posterior wall of the pharynx" through the gating mechanism, the weight of the follicle region feature matrix in the CT image is 0.75, and the weight of other regions is below 0.2, and after adjustment, the correlation strength of the image feature and the text feature is improved, and finally the feature tensor output by the fusion model contains the key information and the interaction relationship of the three types of features.

[0041] Step S132, inputting the physiological feature sequence, the image feature matrix and the text feature vector into the feature fusion model to obtain a rehabilitation feature tensor.

[0042] For example, taking patient A as an example, the physiological feature sequence, image feature matrix and text feature vector of patient A are input into the feature fusion model. The physiological feature sequence includes 128-dimensional time sequence features such as a 0.8mV swallowing peak value of a laryngeal muscle signal in 500ms and a 5μV peak value of 15ms of an auditory potential at 2000Hz stimulation; the image feature matrix includes 64x64-dimensional spatial features such as shape features of a three-dimensional coordinate (38, 42, 18) of a postpharyngeal wall lymphoid follicle in CT and RGB features of a vocal cord hyperemia area (220, 310) in an endoscope; and the text feature vector is 128-dimensional semantic features extracted from “vocal cord mild hyperemia, postpharyngeal wall visible lymphoid follicle hyperplasia”, wherein the corresponding dimensional feature values of “vocal cord hyperemia” and “lymphoid follicle hyperplasia” are 0.85 and 0.78 respectively, and a 256-dimensional rehabilitation feature tensor is obtained.

[0043] It can be understood that the ear-nose-throat pathological data is collected through step S1, the physiological signals, imageology, clinical text and historical rehabilitation data of the patient are comprehensively acquired, and various original inputs are provided for subsequent analysis, so that the one-sidedness of the scheme caused by data loss is avoided; the data quality is improved by removing noise, standardizing data and extracting key features through data preprocessing, so that different types of data have comparability and fusibility, and a foundation is laid for feature fusion; the multi-modal feature fusion realizes cross-modal interaction through an attention mechanism, generates a fusion feature tensor, integrates the advantages of various data, can more comprehensively represent the state of the patient, avoids the limitation of single modal data, and provides feature input for subsequent scheme generation.

[0044] In step S2, the rehabilitation feature tensor is analyzed based on a rehabilitation decision mechanism to obtain an initial rehabilitation scheme.

[0045] In this embodiment, step S2 includes: In step S21, the rehabilitation feature tensor is mapped into a multi-dimensional state feature vector, and the multi-dimensional state feature vector is input into a state space to generate a decision scheme.

[0046] The state space contains 120 action spaces, and the 120 action spaces are obtained by Cartesian product operation based on five rehabilitation training intensities, three training frequencies and eight types of training actions.

[0047] Specifically, the high-dimensional feature tensor is converted into a low-dimensional vector suitable for the state space, and then input into the state space. Each action corresponds to a specific parameter combination by Cartesian product operation, and the optimal decision scheme is selected from the action space by a reinforcement learning algorithm.

[0048] It should be noted that the five rehabilitation training intensities include: low level, mild stimulation, suitable for initial adaptation; low-middle level, moderate stimulation, for function improvement; middle level, increasing intensity, intensive training; high-middle level, high frequency repetition, consolidate effect; senior, personalized adjustment, aiming at tolerance difference. Three training frequencies include: once a day, basic maintenance; twice a day, regular rehabilitation; three times a day, intensive intervention. Eight kinds of training action combinations include: swallowing function, swallowing training, ice stimulation training; hearing recovery, frequency gradient training, sound source positioning training; nasal ventilation, resistance breathing training, alternating nostril breathing training; combine the training contents in swallowing function, hearing recovery and nasal ventilation to obtain eight kinds of training actions.

[0049] Further, in the Cartesian product operation process, each ordered pair is regarded as a tuple, and each element in the tuple is called a component. If the sets A and B are finite sets, their cardinalities are m and n respectively, then the cardinality of the Cartesian product of the sets A and B will be m times n. If set A has m elements and set B has n elements, then their Cartesian product will have m*n elements. In this embodiment, there are three sets of rehabilitation training intensity, training frequency and training action combination, and the corresponding elements are 5, 3 and 8 respectively. Through Cartesian product operation, 120 kinds of action space can be obtained. And through reinforcement learning algorithm, the most suitable decision scheme for the patient is obtained from the 120 kinds of action space.

[0050] In some possible embodiments, again taking patient A as an example, the rehabilitation feature tensor of patient A is mapped to obtain a 128-dimensional state feature vector, which contains information such as a laryngeal myoelectric signal deviation of 0.3 and a vocal cord hyperemia degree of 0.6. After inputting the state space, the optimal decision scheme is screened out based on the current state of the patient, which is middle level intensity, twice a day, combined with ice stimulation training + frequency gradient training + resistance breathing training, forming a decision scheme suitable for patient A.

[0051] Step S22, obtaining the historical patient records of the same case as the current patient, and obtaining the average rehabilitation effect improvement rate of the historical patient records corresponding to the patient using the current decision scheme, the matching degree of the current decision scheme and the current patient, and the safety score of the current scheme, and constructing a reward function, and outputting the safety score of the current patient using the current decision scheme.

[0052] Specifically, the historical patients of the same case are obtained based on the current patient's ear-nose-throat pathological data, including physiological signals, image features, and clinical text descriptions. The k-nearest neighbor algorithm is used to select the top 50 cases with the highest feature similarity. The rehabilitation effect data of these historical cases using the current decision scheme is extracted, and the average rehabilitation effect improvement rate is calculated. The cosine similarity is used to calculate the matching degree of the current decision scheme and the current patient's features, and the degree of fit between the scheme parameters and the patient's physiological and anatomical features is quantified. The safety score of the current scheme is evaluated according to the clinical contraindication rule library, and whether the scheme parameters exceed the patient's tolerance threshold is investigated. The safety score is obtained by inverse conversion of the Risk value. The weight coefficients of the three indicators, i.e., the average rehabilitation effect improvement rate of the historical patients using the current decision scheme, the matching degree of the current decision scheme and the current patient, and the safety score of the current scheme, are set. The reward function is constructed by multiplying the sum of the three indicators and their coefficients, and the safety score of the current decision scheme is output through the reward function.

[0053] It should be noted that the K-nearest neighbor algorithm is to give a training data set and find the K nearest neighbors of the new input instance in the training data set, that is, the K neighbors mentioned above. Most of the K instances belong to a certain class, and the input instance is classified into this class.

[0054] In some possible embodiments, taking ear-nose-throat patient A as an example, 50 historical patients of the same case are selected by the k-nearest neighbor algorithm, of which 12 use the same scheme as the current scheme, such as intermediate intensity + 2 times a day + ice stimulation + frequency gradient training. The average rehabilitation effect improvement rate is 0.8 per day, the similarity between the current scheme and the patient's features is 0.85, the training intensity of the scheme does not exceed the patient's laryngeal muscle safety threshold, the Risk value is 0.15, the safety score is 0.85, and the weight coefficients of the three indicators are assumed to be 0.5, 0.3, and 0.2, respectively. The safety score of the current decision is 0.825 by substituting into the reward function.

[0055] It can be understood that the Risk value mentioned above represents the relative risk, and in this embodiment, the safety score can be obtained through the relative risk. If the obtained relative risk is 0.74, the safety score is 0.26.

[0056] Step S23, extract the three decision schemes with the highest safety scores from all decision schemes, and form an initial rehabilitation scheme.

[0057] In some possible embodiments, taking patient A as an example, the top three of the safety scores in the several decision schemes are as follows: scheme 1: low-to-moderate intensity + once a day + empty swallowing training + frequency gradient training, with a safety score of 0.95, which is suitable for initial adaptation, low training intensity and low frequency, and avoids high-load actions on the larynx; scheme 2: moderate intensity + twice a day + ice stimulation training + resistance breathing training, with a safety score of 0.92, which is moderate in intensity, gentle in ice stimulation action, and the breathing training pressure is within a safe range; and scheme 2: low-to-moderate intensity + twice a day + vocal coordination training + alternating nostril breathing training, with a safety score of 0.90, which focuses on functional coordination and the combination of actions has no contraindication risk.

[0058] It can be understood that the initial rehabilitation scheme is generated by fusing the feature tensor to construct the deep reinforcement learning model in step S2, the state space and the action space are adapted through the feature dimension, and the historical data and the individual characteristics of the patient are combined through the reward function to generate the initial scheme, which combines the historical data and takes into account the uniqueness of the patient, provides targeted input for subsequent optimization, and ensures the effectiveness and feasibility of the scheme through the training of the policy network.

[0059] In step S3, a rehabilitation prediction model is constructed based on the rehabilitation database, the initial rehabilitation scheme is input into the rehabilitation prediction model and is optimized to obtain an optimized rehabilitation scheme.

[0060] In this embodiment, step S3 includes: In step S31, a rehabilitation prediction model is constructed based on a gradient boosting decision tree and a recurrent neural network.

[0061] The gradient boosting decision tree sorts the feature importance through the regression tree, and screens out the key features with a weight of the top 30% on the rehabilitation effect, and the recurrent neural network obtains the time sequence dependence of the rehabilitation state through the gating mechanism.

[0062] Specifically, the key features with a weight of the top 30% on the rehabilitation effect are screened out from the physiological feature sequence, the image feature matrix and the text feature vector, such as training intensity, laryngeal muscle amplitude, lesion shrinkage rate, etc.; and the recurrent neural network adopts a structure with a gating mechanism, such as a long short-term memory or a gated recurrent unit, the gating unit selectively retains or forgets historical information through the synergistic action of the input gate, the forget gate and the output gate, and captures the time sequence dependence of the rehabilitation state, such as the influence of the training intensity of the previous week on the functional recovery of the next week, and the screened key features are input into the recurrent neural network in time sequence to output the rehabilitation state prediction value in the future period of time.

[0063] It should be noted that the gradient boosting decision tree is integrated by multiple regression trees, each tree is trained based on the residual of the previous tree, and the feature importance is sorted by calculating the splitting contribution of the feature on each tree.

[0064] It should be noted that the top 30% of the key features can generally include more than 80% of the contribution to the rehabilitation effect, so obtaining the data of 30% avoids retaining too many low-weight features and ensures that important information is not lost.

[0065] In step S32, the initial rehabilitation scheme is input into the rehabilitation prediction model to obtain a predicted rehabilitation state curve.

[0066] Specifically, the parameters in the scheme are converted into a feature vector recognizable by the model, combined with the key features screened by the gradient boosting decision tree, and formed into time series input data. The training parameters of the initial scheme are arranged on the time axis every week as input into the recurrent neural network. The gate unit dynamically adjusts the information retention degree according to the historical rehabilitation law, and outputs the rehabilitation state prediction value of each week in the future. A right-angle coordinate system of rehabilitation state prediction value and time is constructed, and all prediction values are connected in time sequence to form a predicted rehabilitation state curve. Through the predicted rehabilitation state curve, the rehabilitation trend of the patient after the implementation of the scheme can be directly observed.

[0067] In some possible embodiments, taking the ear-nose-throat patient A as an example, the gradient boosting decision tree trains the 128-dimensional features of the patient, calculates the feature importance through 100 regression trees, and screens out the top 30% of the key features such as training intensity, laryngeal electromyography peak value, vocal cord hyperemia area, and “smoking history” text features. These features are constructed into time series data in units of every week, such as training intensity at the intermediate level and electromyography peak value at 0.8 mV in the first week, training intensity at the intermediate level and electromyography peak value at 0.7 mV in the second week, etc. The time series data is input into the recurrent neural network with a gated recurrent unit. The forget gate ignores irrelevant short-term fluctuations, and the input gate and the output gate focus on retaining the long-term correlation between the training intensity and the electromyography changes. Finally, the model predicts that the vocal cord hyperemia area of the patient will decrease by 20% and the auditory evoked potential threshold will decrease by 5 dB in the third week. The error with respect to the actual rehabilitation trend is controlled within a certain range.

[0068] In step S33, the initial rehabilitation scheme is optimized based on the predicted rehabilitation state curve using an improved genetic algorithm to obtain an optimized rehabilitation scheme.

[0069] In this embodiment, step S33 includes: In step S331, the training parameters in the initial rehabilitation scheme are encoded using real number coding chromosomes based on the improved genetic algorithm, and three objective functions of rehabilitation period, training pain, and functional recovery are set, and patient adaptation constraints are added.

[0070] The constraint condition is that the training intensity of the patient does not exceed the tolerance threshold, and the training frequency of the patient conforms to the physiological rhythm of the human body.

[0071] Specifically, the training parameters in the initial rehabilitation scheme are coded with real numbers to construct a chromosome, such as rehabilitation training intensity, training frequency, training action combination, etc., and each gene corresponds to a specific numerical value of a parameter, such as training intensity coded as a continuous value of 0.1-1.0, corresponding to five levels of intensity, to ensure the accuracy of the parameters and the continuity of the search space; at the same time, three objective functions are set, respectively, to minimize the rehabilitation period, specifically by predicting the number of days to reach the target state, to minimize the training pain, specifically by quantifying the patient's discomfort through visual analog scale, and to maximize the functional recovery, taking the predicted functional recovery rate as an indicator, and obtaining the optimal solution among the three objective functions through multi-objective optimization; at the same time, constraint conditions are added, such as training intensity not exceeding the patient's tolerance threshold, such as the intensity level corresponding to the maximum tolerance value of the laryngeal electromyogram, and training frequency complying with the human physiological rhythm, such as the interval between two hearing training not less than 6 hours, to avoid the scheme exceeding the safety range or violating the physiological law.

[0072] In some possible embodiments, also taking the ear-nose-throat patient A as an example, the parameters such as "training intensity, daily training frequency, and ice stimulation duration" in the scheme are coded with real numbers, and the chromosome can be represented as [0.6, 2, 15], where 0.6 corresponds to medium intensity, 2 represents 2 times a day, and 15 is 15 minutes of ice stimulation each time; the three objective functions can be set as: rehabilitation period less than or equal to 21 days, training pain score less than or equal to 3 points, using the ten-point system, and functional recovery rate greater than or equal to 60%; the constraint condition is that the training intensity does not exceed the patient's laryngeal electromyogram tolerance value of 0.7, corresponding to level 7, and the daily training interval is less than 6 hours.

[0073] Step S332, iteratively optimizing the initial rehabilitation scheme based on the adaptive crossover probability and the elite preservation strategy to obtain an optimized rehabilitation scheme.

[0074] Specifically, in the iterative optimization process, the improved genetic algorithm first codes the training parameters of the initial rehabilitation scheme with real numbers to form an initial population, calculates the fitness of each individual through the three objective functions of rehabilitation period, training pain, and functional recovery, generates a new population through selection, crossover, and mutation operations, continuously evolves through the adaptive crossover probability and the elite preservation strategy, until the population converges, and finally selects the scheme with the best comprehensive performance from the optimal solution set as the optimized rehabilitation scheme.

[0075] It should be noted that the traditional genetic algorithm adopts fixed crossover probability and mutation probability, which often causes problems of insufficient population diversity and excessive convergence; at the same time, the traditional algorithm is simple in processing of constraint conditions, so that the final obtained result is an invalid result. In the embodiment, the improved genetic algorithm introduces adaptive crossover probability, which is dynamically adjusted according to the individual fitness, and low crossover probability is used to reserve excellent parameters for high-quality schemes, and high crossover probability is used to promote optimization for poor-quality schemes; at the same time, the elite reservation strategy is introduced, which directly reserves the optimal individual of each generation to ensure that the high-quality scheme is not lost and to speed up the convergence; the constraint condition is strengthened, and the scheme that violates the physiological rhythm and exceeds the limit of intensity is constrained to ensure that the generated scheme is feasible.

[0076] It should be noted that the adaptive crossover probability is dynamically adjusted according to the fitness of individuals in the population. For individuals with high fitness, a lower crossover probability is used to reserve excellent genes, and for individuals with low fitness, a higher crossover probability is used to promote gene recombination and increase population diversity; the elite reservation strategy is to directly reserve the top 10% of individuals in terms of fitness to the next generation in each generation evolution, avoiding the loss of excellent genes due to crossover and mutation operations.

[0077] In some possible embodiments, still taking the ear-nose-throat patient A as an example, the initial population obtained includes parameter encoding of 3 initial schemes, such as scheme 1 encoding [0.6 (medium intensity), 2 (2 times a day), 15 (ice stimulation duration 15 minutes)], after calculating the fitness, for individuals with high fitness, such as scheme 1 with fitness 0.75, a crossover probability of 0.6 is used, and for individuals with low fitness, a crossover probability of 0.9 is used, the top 10% of elite individuals are directly reserved to the next generation in each generation, after multiple iterations, the population gradually converges, and the final optimized scheme is encoded as [0.55 (medium intensity), 2 (2 times a day), 12 (ice stimulation duration 12 minutes)].

[0078] It can be understood that the initial rehabilitation scheme is optimized and adjusted through step S3, the rehabilitation state prediction model is constructed to predict the implementation effect of the scheme, the improved genetic algorithm is used to obtain the optimal solution among multiple targets, so that the scheme is better in terms of rehabilitation period, training pain and functional recovery, while the constraint condition is followed to ensure the safety of the scheme, so that the optimized rehabilitation scheme is more in line with the needs of the patient, and the practicality and scientificity of the scheme are improved.

[0079] Step S4, the optimized rehabilitation scheme is evaluated professionally, and a rehabilitation feedback mechanism is executed based on the evaluation result.

[0080] The ear-nose-throat quantitative evaluation index is constructed based on the ear-nose-throat quantitative evaluation index, the optimized rehabilitation scheme is scored based on the ear-nose-throat quantitative evaluation index, all sub-index scores of the ear-nose-throat quantitative evaluation index are obtained, the mean value of the sub-index scores is calculated, the mean value threshold and the sub-index score threshold are set, and the mean value and the sub-index score are compared with the corresponding threshold value respectively: When the mean value is greater than the mean value threshold and all sub-index scores are greater than the sub-index score threshold, it is determined that the evaluation passes, and the optimized rehabilitation scheme is used to assist the patient in treatment; When the mean value and all sub-index scores are less than or equal to the corresponding threshold value, it is determined that the evaluation does not pass, external feedback is obtained, the external feedback is converted into a quantitative constraint condition, and the value step S2 is returned.

[0081] Specifically, the scheme adaptation degree includes a physiological feature matching degree, such as a training intensity adaptation ratio with an electromyographic threshold value, an anatomic structure matching degree, such as a training action corresponding degree with a lesion position, a medical history matching degree, such as a scheme avoidance rate for previous medical history, and the like. Each sub-index is scored by 0-10 points and the mean value is taken; the implementation safety includes a training parameter safety redundancy, such as a difference ratio between an actual intensity and a safety threshold value, a high-risk action avoidance rate, and an emergency plan completeness, which are also scored by 0-10 points and the mean value is taken; the expected effect rationality includes a consistency with a historical case effect, that is, a deviation rate of a predicted value from an actual mean value, a functional recovery target achievement probability, and a rehabilitation cycle rationality, which are scored by 0-10 points and the mean value is taken. The doctor scores the optimized rehabilitation scheme according to the above sub-indexes, and calculates the mean value of the ear-nose-throat quantitative evaluation index score, and sets the mean value threshold and the sub-index threshold value, wherein the ear-nose-throat quantitative evaluation index includes the scheme adaptation degree, the implementation safety, and the expected effect rationality.

[0082] Further, the external feedback opinion is expressed as, for example, a doctor's feedback opinion on the optimized rehabilitation index. When the mean value is greater than the mean value threshold and no sub-index score is less than the sub-index threshold value, it is determined to pass, and the doctor can refer to the scheme to assist in treatment; when the mean value is greater than the mean value threshold or there is a sub-index less than the sub-index threshold value, it is determined to not pass, and the doctor's opinion needs to be collected, such as “training intensity needs to be reduced”, and the doctor's opinion is converted into a quantitative constraint condition, such as setting the upper limit of the intensity to 90% of the original threshold value and returning to step S2 for re-processing of data and subsequent optimization.

[0083] In some possible embodiments, still taking the ear-nose-throat patient A as an example, the three indicator scores of the scheme fitness are 8.5, 7.8 and 8.2 respectively, and the average is 8.17; the indicator scores of the implementation safety are 9.0, 8.5 and 8.0 respectively, and the average is 8.5; the indicator scores of the rationality of the expected effect are 7.5, 8.2 and 7.8 respectively, and the average is 7.83. The average threshold can be set as 8.5, and the threshold of the sub-indicators is 8, which is determined as not passing. The opinion of the doctor that the safety needs to be improved is collected, and is converted into a quantitative constraint condition that the upper limit of the training intensity is reduced by 15%. The data is reprocessed in step S2, the rehabilitation scheme is iteratively optimized, and finally the optimized rehabilitation scheme is output as an auxiliary tool for the doctor to determine the final treatment scheme in combination with the clinical judgment of the doctor.

[0084] It can be understood that the optimized scheme is verified in step S4, details that may be ignored by the algorithm are found with the help of the clinical experience of the doctor, and the fitness, safety and rationality of the expected effect of the scheme are ensured. Meanwhile, the rehabilitation scheme is iteratively optimized according to the feedback of the doctor until the evaluation is passed, and finally the optimized rehabilitation scheme is output as an auxiliary tool for the doctor to determine the final treatment scheme in combination with the clinical judgment of the doctor, which not only plays the advantages of the algorithm, but also embodies the leading role of the clinical decision, and ensures the reliability and applicability of the scheme in practical application.

[0085] Figure 3 A schematic diagram of a multi-modal ear-nose-throat patient rehabilitation scheme optimization system is shown, which can implement the idea of the present application.

[0086] Specifically, a multi-modal ear-nose-throat patient rehabilitation scheme optimization system comprises: An acquisition module is configured to acquire ear-nose-throat pathological data of a patient, input an initial rehabilitation scheme into a rehabilitation prediction model and optimize the initial rehabilitation scheme to obtain an optimized rehabilitation scheme.

[0087] A processing module is configured to preprocess the ear-nose-throat pathological data.

[0088] A fusion module is configured to fuse the preprocessed ear-nose-throat pathological data to obtain a rehabilitation feature tensor.

[0089] A construction module is configured to construct a rehabilitation prediction model based on a rehabilitation database.

[0090] An analysis module is configured to analyze the rehabilitation feature tensor based on a rehabilitation decision mechanism to obtain an initial rehabilitation scheme.

[0091] An evaluation module performs professional evaluation on the optimized rehabilitation scheme, and executes a rehabilitation feedback mechanism based on the evaluation result.

[0092] The specific use mode and role of the embodiment are described below: First, ear-nose-throat pathological data collection and fusion are performed through step S1, physiological signals, image features and clinical text data of the patient are comprehensively integrated, the limitation of a single data type is broken, the generated scheme is more in line with the individual characteristics of the patient, and deviation of the scheme caused by individual differences is avoided; then, the rehabilitation prediction model constructed through steps S2 and S3 is combined with the improved genetic algorithm for iterative optimization, the rehabilitation effect is accurately predicted, and the training parameters are dynamically adjusted, so that the optimal balance between the rehabilitation period, training pain and functional recovery is achieved, and the accuracy and effectiveness of the rehabilitation treatment are improved; finally, the quantitative evaluation index is combined with the doctor evaluation through step S4, the scientificity of the scheme is judged through the set mean threshold value, the role of the clinical experience of the doctor is played, the scheme is continuously optimized through the feedback iteration of the quantitative constraint condition, the treatment risk is reduced, and a safer and more efficient auxiliary rehabilitation scheme is provided for the patient.

[0093] In addition, the embodiment of the present application further provides an electronic device, comprising: At least one processor; and a memory connected with the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method proposed in the embodiment one of the present application.

[0094] The various constituent components of the electronic device are specifically introduced as follows: The processor is the control center of the electronic device, and can be one processor or a plurality of processing elements. For example, the processor is one or more central processing units (CPU), or is one or more application specific integrated circuits (ASIC), or is one or more integrated circuits configured to implement the embodiment one of the present application, such as one or more digital signal processors (DSP), or one or more field programmable gate arrays (FPGA).

[0095] The processor can perform various functions of the electronic device by running or executing a software program stored in the memory, and calling data stored in the memory.

[0096] The memory is used to store a software program for implementing the scheme of the present application, and is controlled by the processor to perform, and the specific implementation can refer to the method embodiments described above, which will not be repeated here.

[0097] The memory can be a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (RAM) or other type of dynamic storage device that can store information and instructions, an electrically erasable programmable read-only memory (EEPROM), an optical disc read-only memory (D-ROM), or other optical disc storage, optical disc storage (including compact discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and capable of being accessed by a computer, but not limited to. The memory can be integrated with the processor or exist independently and be coupled with the processor through the interface circuit of the electronic device, and the embodiments of the present application do not make specific limitations thereto.

[0098] The above embodiments can be implemented in whole or in part by software, hardware (such as a circuit), firmware, or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the flow or function described in the embodiments of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another, for example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center by limited (for example, infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. containing a set of one or more available media. The available media can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state disk.

[0099] It should be understood that the term "and / or" in this text only describes the association relationship of the associated objects, which means that there can be three relationships, for example, and / or I frame / P frame interval offset rate, which means that I frame / P frame interval offset rate alone, I frame / P frame interval offset rate and I frame / P frame interval offset rate exist at the same time, and I frame / P frame interval offset rate alone, where I frame / P frame interval offset rate can be singular or plural. In addition, the character " / " in this text generally indicates that the associated objects before and after are an "or" relationship, but it can also mean an "and / or" relationship, which can be understood in the context.

[0100] It should be understood that in the embodiments of the present application, the size of the sequence number of the above-mentioned processes does not mean the order of execution, and the execution order of the processes should be determined according to its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0101] The above-described embodiments are only used to illustrate the technical solutions of the present application, but not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can still be modified, or some technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A multimodal ear, nose and throat patient rehabilitation protocol optimization method, characterized in that, The method comprises: S1: obtaining ear-nose-throat pathological data of a patient, and performing a preprocessing operation on the ear-nose-throat pathological data, fusing features of the preprocessed ear-nose-throat pathological data, and obtaining a rehabilitation feature tensor; S2: analyzing the rehabilitation feature tensor based on a rehabilitation decision mechanism, and obtaining an initial rehabilitation scheme; S3: constructing a rehabilitation prediction model based on a rehabilitation database, inputting the initial rehabilitation scheme into the rehabilitation prediction model and optimizing the initial rehabilitation scheme, and obtaining an optimized rehabilitation scheme; S4: performing professional evaluation on the optimized rehabilitation scheme, and executing a rehabilitation feedback mechanism based on the evaluation result.

2. The method of claim 1, wherein, Obtaining ear-nose-throat pathological data of a patient, and performing a preprocessing operation on the ear-nose-throat pathological data, fusing features of the preprocessed ear-nose-throat pathological data, and obtaining a rehabilitation feature tensor, comprising: The ear-nose-throat pathological data comprises physiological signal data, imaging data and clinical text data of the patient; Performing a data preprocessing operation on the physiological signal data, imaging data and clinical text data of the patient respectively, obtaining a physiological feature sequence, an imaging feature matrix and a text feature vector; Constructing a feature fusion model, inputting the physiological feature sequence, the imaging feature matrix and the text feature vector into the feature fusion model, and obtaining a rehabilitation feature tensor.

3. The method of claim 2, wherein the method is used to optimize a multi-modal otolaryngology patient rehabilitation regimen. Performing a data preprocessing operation on the physiological signal data, imaging data and clinical text data of the patient respectively, obtaining a physiological feature sequence, an imaging feature matrix and a text feature vector, comprising: Removing baseline drift and high-frequency noise in the physiological signal data based on wavelet transform, eliminating motion artifacts and power frequency interference based on adaptive filtering operation, obtaining preliminary physiological signals, segmenting the preliminary physiological signals based on resolution, and extracting features from each segmented signal, obtaining a physiological feature sequence; Extracting a region of interest in the imaging data based on a U-net model, and performing feature point matching on the region of interest based on a scale-invariant feature transform algorithm, generating an imaging feature matrix; Performing word segmentation and part-of-speech tagging on the clinical text data based on natural language processing, and performing semantic encoding on the clinical text data based on a bidirectional long short-term memory network, obtaining a text feature vector; Performing multiple alignments of the physiological feature sequence, the imaging feature matrix and the text feature vector based on multi-dimensional collaborative processing; The multi-dimensional collaborative processing comprises establishing a time synchronization relationship between the physiological feature sequence and the imaging feature matrix based on a dynamic time warping algorithm, spatially registering three-dimensional coordinates of all imaging feature matrices based on a set projection model, and constructing a semantic association relationship between the text feature vector and the imaging feature matrix.

4. The method of claim 2, wherein the method is used for optimizing a multi-modal ear, nose, and throat patient rehabilitation protocol. Constructing a feature fusion model, comprising: Constructing a feature fusion model based on an ear-nose-throat interaction mechanism composed of time-space interaction, time-language interaction and space-language interaction; The time-space interaction is represented by projecting the physiological feature sequence into the imaging feature matrix based on a variable convolution, the time-language interaction is represented by labeling the text feature in the physiological feature sequence based on cross-modal attention, and the space-language interaction is represented by adjusting the association strength between the imaging feature matrix and the text feature dimension based on a gating mechanism.

5. The multi-modal ear-nose-throat patient rehabilitation protocol optimization method of claim 1, wherein, Based on the rehabilitation decision mechanism, the rehabilitation feature tensor is analyzed to obtain an initial rehabilitation scheme, including: The rehabilitation feature tensor is mapped to a multi-dimensional state feature vector, and the multi-dimensional state feature vector is input into a state space to generate a decision scheme; The state space contains 120 action spaces, which are obtained by Cartesian product operation based on five rehabilitation training intensities, three training frequencies, and eight types of training actions; The historical patient records of the same case as the current patient are obtained, and the average rehabilitation effect improvement rate of the current decision scheme, the matching degree of the current decision scheme and the current patient, and the safety score of the current scheme are obtained, and a reward function is constructed to output the safety score of the current patient using the current decision scheme. The three decision schemes with the highest safety score in all decision schemes are extracted and combined to form an initial rehabilitation scheme.

6. The method of claim 1, wherein: Based on the rehabilitation database, a rehabilitation prediction model is constructed, and the initial rehabilitation scheme is input into the rehabilitation prediction model for optimization to obtain an optimized rehabilitation scheme, including: The rehabilitation prediction model is constructed based on gradient boosting decision tree and recurrent neural network; The gradient boosting decision tree sorts the feature importance by regression tree, and selects the key features with the top 30% weight affecting the rehabilitation effect, and the recurrent neural network obtains the time sequence dependence of the rehabilitation state through the gating mechanism; The initial rehabilitation scheme is input into the rehabilitation prediction model to obtain a predicted rehabilitation state curve; Based on the predicted rehabilitation state curve, the initial rehabilitation scheme is optimized using an improved genetic algorithm to obtain an optimized rehabilitation scheme.

7. The method of claim 6, wherein the method is used to optimize a multi-modal otolaryngology patient rehabilitation regimen. Based on the predicted rehabilitation state curve, the initial rehabilitation scheme is optimized using an improved genetic algorithm to obtain an optimized rehabilitation scheme, including: The training parameters in the initial rehabilitation scheme are encoded using real number coding chromosomes based on the improved genetic algorithm, and three objective functions of rehabilitation period, training pain and functional recovery are set, and patient adaptation constraints are added; The initial rehabilitation scheme is iteratively optimized based on adaptive crossover probability and elite reservation strategy to obtain an optimized rehabilitation scheme.

8. The multi-modal ear-nose-throat patient rehabilitation protocol optimization method of claim 1, wherein, The optimized rehabilitation scheme is evaluated professionally, and a rehabilitation feedback mechanism is executed based on the evaluation results, including: Ear-nose-throat quantitative evaluation indexes of scheme adaptation, implementation safety and rationality of expected effect are constructed, the optimized rehabilitation scheme is scored based on the ear-nose-throat quantitative evaluation indexes, all sub-index scores of the ear-nose-throat quantitative evaluation indexes are obtained, and the mean value of the sub-index scores is calculated, the mean value threshold and the sub-index score threshold are set, and the mean value and the sub-index score are compared with the corresponding threshold respectively: When the mean value is greater than the mean value threshold and all sub-index scores are greater than the sub-index score threshold, it is determined that the evaluation is passed, and the patient is treated based on the optimized rehabilitation scheme; When the mean value and all sub-index scores are less than or equal to the corresponding threshold, it is determined that the evaluation is not passed, external feedback is obtained, the external feedback is converted into a quantitative constraint condition and returned to step S2.

9. A multi-modal ear, nose, and throat patient rehabilitation protocol optimization system, comprising: including: The acquisition module is used for acquiring ear-nose-throat pathological data of a patient, inputting an initial rehabilitation scheme into a rehabilitation prediction model and optimizing the rehabilitation prediction model, and acquiring an optimized rehabilitation scheme; The processing module is used for preprocessing the ear-nose-throat pathological data; The fusion module is used for performing feature fusion on the preprocessed ear-nose-throat pathological data, and acquiring a rehabilitation feature tensor; The construction module is used for constructing the rehabilitation prediction model based on a rehabilitation database; The analysis module is used for analyzing the rehabilitation feature tensor based on a rehabilitation decision mechanism, and acquiring the initial rehabilitation scheme; The evaluation module is used for performing professional evaluation on the optimized rehabilitation scheme, and executing a rehabilitation feedback mechanism based on an evaluation result.

Images