Intelligent diagnosis and monitoring system for otolaryngology department based on multi-modal data fusion

Abstract

The invention provides an ear-nose-throat department intelligent diagnosis and monitoring system based on multi-modal data fusion, and relates to the technical field of multi-modal data fusion. Multi-modal monitoring data is acquired, and an endoscope monitoring image of a patient is extracted; performing edge monitoring on the endoscope monitoring image to obtain an endoscope monitoring edge image, generating a secretion disturbance mask according to the endoscope monitoring edge image, and determining an endoscope monitoring restoration image according to the secretion disturbance mask and the endoscope monitoring image; determining a pixel decision window of each pixel point in the endoscope monitoring restored image, and performing information decision based on the corresponding pixel decision window to obtain an endoscope monitoring decision image; and extracting monitoring text data, and performing feature fusion on the monitoring text data and the endoscope monitoring decision image to generate a fusion association map. According to the method, the image quality of the endoscope monitoring image can be improved, and the correlation map of the patient is generated in combination with the monitoring text, so that knowledge fusion in ear-nose-throat monitoring is realized.

Description

Technical Field

[0001] The present application relates to the technical field of multimodal data fusion, and more specifically, to an intelligent otolaryngology diagnosis and monitoring system based on multimodal data fusion. Background Art

[0002] Intelligent diagnosis and monitoring of otolaryngology based on multimodal data fusion aims to build a comprehensive, accurate and efficient intelligent medical system by integrating multi-source heterogeneous data such as nasopharyngeal endoscopic images, voice audio, physiological monitoring signals, and structured medical record texts, so as to achieve auxiliary diagnosis of common otolaryngological diseases and early lesions, identification of disease evolution trends, and continuous health monitoring. This method not only breaks through the limitations of a single modality in information expression, but also relies on the deep combination of artificial intelligence and medical knowledge to enhance the understanding of complex lesion morphology, symptom interaction, and spatiotemporal evolution processes. The overall process of the system includes multimodal data acquisition, feature alignment and semantic fusion between modalities, disease identification and severity assessment driven by fusion features, health status monitoring based on graphs and time series models, and a visual feedback and intelligent interaction platform for doctors and patients. Especially in the diagnosis and treatment of diseases in the nasopharynx, throat and other parts, multimodal collaboration has natural advantages: images can directly present the morphology of lesions, such as swelling, inflammation, lumps, etc.; voice data indirectly reflects the functional status of the vocal cords or throat, such as hoarseness, abnormal pitch, etc.; physiological signal records reflect potential systemic symptoms such as sleep breathing and heart rate, while text medical records supplement unstructured knowledge such as medical history and drug use.

[0003] However, in the prior art, various modalities are distributed differently in physical space and semantic space, making it impossible to achieve effective cross-modal alignment and build a unified semantic expression space. In addition, the endoscopic monitoring images of the nasopharynx have disturbances such as uneven lighting, reflections, highlight mucus occlusion, dynamic blur, and non-structural sound interference such as coughing and swallowing during the pronunciation of patients, which seriously affect the data quality. Therefore, how to improve the image quality of endoscopic monitoring images and generate a patient association map in combination with monitoring texts to achieve knowledge fusion in ENT monitoring is a difficult problem faced by the industry. Summary of the invention

[0004] The present application provides an intelligent ENT diagnosis and monitoring system based on multimodal data fusion, which can improve the image quality of endoscopic monitoring images and generate a patient association map in combination with monitoring text to achieve knowledge fusion in ENT monitoring.

[0005] The present application provides an intelligent otolaryngology diagnosis and monitoring system based on multimodal data fusion, the diagnosis and monitoring system comprising: An image acquisition module, used to acquire multimodal monitoring data of a patient in ENT monitoring, and extract an endoscopic monitoring image of the patient from the multimodal monitoring data; an image restoration module, configured to perform edge monitoring on the endoscopic monitoring image to obtain an endoscopic monitoring edge image, generate a secretion disturbance mask based on the endoscopic monitoring edge image, and determine an endoscopic monitoring restoration image of the patient based on the secretion disturbance mask and the endoscopic monitoring image; An information decision module is used to determine a pixel decision window for each pixel point in the endoscopic monitoring restoration image, and to make information decisions on each pixel point in the endoscopic monitoring restoration image based on the corresponding pixel decision window, thereby obtaining an endoscopic monitoring decision image of the patient; The association map generation module is used to extract the patient's monitoring text data from the multimodal monitoring data, perform feature fusion on the monitoring text data and the endoscopic monitoring decision image, and then generate a fused association map of the patient in ENT monitoring.

[0006] In this embodiment, the multimodal monitoring data includes endoscopic monitoring images and monitoring text data of the patient during ENT monitoring.

[0007] In this embodiment, the Canny edge detection algorithm is used to perform edge detection on the endoscope monitoring image.

[0008] In this embodiment, generating a secretion disturbance mask based on the endoscope monitoring edge image specifically includes: Performing morphological processing on the endoscope monitoring edge image to obtain a closed edge image; Filling holes in the closed edge image to obtain a simply connected region image; The singly connected region image is subjected to corrosion processing, and a secretion disturbance mask is generated according to the corrosion processing result.

[0009] In this embodiment, determining the patient's endoscopic monitoring restoration image based on the secretion disturbance mask and the endoscopic monitoring image is performed by performing an XOR operation on the secretion disturbance mask and the endoscopic monitoring image to obtain the patient's endoscopic monitoring restoration image.

[0010] In this embodiment, determining the pixel decision window of each pixel point in the endoscope monitoring restoration image specifically includes: Get the preset pixel window; For each pixel point in the endoscope monitoring restoration image, determining the information fineness corresponding to the pixel point according to the pixel window; The pixel decision window of the pixel point is determined based on the information subtlety, and then the pixel decision window of each pixel point in the endoscope monitoring restoration image is obtained.

[0011] In this embodiment, determining the information fineness corresponding to the pixel point according to the pixel window specifically includes: Taking the pixel point as the window center of the pixel window, and then determining all the neighborhood windows of the pixel window; Determining a pixel heterogeneity coefficient between the pixel window and each neighborhood window; The information fineness corresponding to the pixel point is determined by all pixel heterogeneity coefficients.

[0012] In this embodiment, determining the pixel decision window of the pixel point based on the information subtlety specifically includes: Get the preset subtlety threshold; When the information subtlety is greater than the subtlety threshold, setting the pixel decision window size of the pixel point to 3 pixels×3 pixels; When the information subtlety is not greater than the subtlety threshold, the pixel decision window size of the pixel point is set to 7 pixels×7 pixels.

[0013] In this embodiment, information decision is made on each pixel point in the endoscopic monitoring restoration image based on the corresponding pixel decision window, and then the endoscopic monitoring decision image of the patient is obtained, which specifically includes: Determine the background noise floor of the endoscope monitoring restored image; For each pixel point in the endoscope monitoring restoration image, the local information complexity and local pixel level of the pixel point are determined according to the corresponding pixel decision window; Performing information decision on the pixel point through the background noise, the local information complexity and the local pixel level to obtain a decision pixel point corresponding to the pixel point, and then obtaining a decision pixel point corresponding to each pixel point in the endoscope monitoring restoration image; The patient's endoscopic monitoring decision image is constructed based on all decision pixels.

[0014] In this embodiment, the monitoring text data and the endoscope monitoring decision image are feature fused to generate a fusion association map of the patient in ENT monitoring, specifically including: Performing semantic analysis on the monitoring text data to obtain structured knowledge of the patient in ENT monitoring; Extracting lesion features from the endoscope monitoring decision image to obtain a lesion image feature vector; The structured knowledge and the lesion image feature vector are jointly modeled to generate a fused association map of the patient in ENT monitoring.

[0015] The technical solution provided by the embodiments disclosed in this application has the following beneficial effects: The multimodal monitoring data of the patient in ear, nose and throat monitoring is acquired through the image acquisition module, and the endoscopic monitoring image of the patient is extracted from the multimodal monitoring data; the image restoration module performs edge monitoring on the endoscopic monitoring image to obtain an endoscopic monitoring edge image, generates a secretion disturbance mask based on the endoscopic monitoring edge image, and determines the patient's endoscopic monitoring restoration image according to the secretion disturbance mask and the endoscopic monitoring image; the information decision module determines the pixel decision window of each pixel point in the endoscopic monitoring restoration image, performs information decision on each pixel point in the endoscopic monitoring restoration image based on the corresponding pixel decision window, and then obtains the patient's endoscopic monitoring decision image; the association map generation module extracts the patient's monitoring text data from the multimodal monitoring data, performs feature fusion on the monitoring text data and the endoscopic monitoring decision image, and then generates a fused association map of the patient in ear, nose and throat monitoring.

[0016] It can be seen that in this application, first, in ENT endoscope monitoring, edge extraction is performed on the endoscopic monitoring image, secretion disturbance mask is generated, and a processing flow of the endoscopic monitoring restoration image is constructed, which can effectively improve the structural clarity and lesion visibility of the image, and significantly suppress image noise and information loss caused by secretions, blur occlusion, etc.; then, a pixel decision window is adaptively determined for each pixel in the endoscopic monitoring restoration image, and local information decisions are made accordingly, that is, a small window is used in detail-rich areas to retain edges and lesion contours, and a large window is used in smooth areas to suppress noise and artifacts, thereby significantly improving the overall image quality and structural clarity; finally, the monitoring text data is feature fused with the endoscopic monitoring decision image to generate a fused association map of the patient in ENT monitoring. By fusing the structured semantic information in the text with the lesion area features in the image, information complementarity and semantic enhancement between multiple modalities can be achieved, which can improve the image comprehension ability and semantic clarity, remove interference areas such as secretions through image decision optimization processing, improve the overall image quality, and realize knowledge fusion in ENT monitoring.

[0017] In summary, the technical solution adopted in this application can improve the image quality of endoscopic monitoring images, and generate a patient association map in combination with the monitoring text to achieve knowledge fusion in ENT monitoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0019] Figure 1 It is a module structure diagram of the otolaryngology intelligent diagnosis and monitoring system based on multimodal data fusion provided by the present application; Figure 2 is an exemplary flow chart for generating a secretion perturbation mask according to the present application; Figure 3 This is an exemplary flow chart for determining the pixel decision window of each pixel point in the endoscopic monitoring restoration image provided by the present application. DETAILED DESCRIPTION

[0020] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0021] The embodiment of the present application provides an otolaryngology intelligent diagnosis and monitoring system based on multimodal data fusion, the core of which is to obtain multimodal monitoring data of patients in otolaryngology monitoring through an image acquisition module, and extract the patient's endoscopic monitoring image from the multimodal monitoring data; the image restoration module performs edge monitoring on the endoscopic monitoring image to obtain an endoscopic monitoring edge image, generates a secretion disturbance mask based on the endoscopic monitoring edge image, and determines the patient's endoscopic monitoring restoration image based on the secretion disturbance mask and the endoscopic monitoring image; the information decision module determines the pixel decision window of each pixel point in the endoscopic monitoring restoration image, performs information decision on each pixel point in the endoscopic monitoring restoration image based on the corresponding pixel decision window, and then obtains the patient's endoscopic monitoring decision image; the association map generation module extracts the patient's monitoring text data from the multimodal monitoring data, performs feature fusion on the monitoring text data and the endoscopic monitoring decision image, and then generates a fused association map of the patient in otolaryngology monitoring. The above scheme can improve the image quality of endoscopic monitoring images and generate a patient association map in combination with monitoring text to achieve knowledge fusion in ENT monitoring.

[0022] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods. Figure 1 As shown, this figure is a module structure diagram of an otolaryngology intelligent diagnosis and monitoring system based on multimodal data fusion according to this embodiment of the present application. The diagnosis and monitoring system includes: an image acquisition module 100, an image restoration module 200, an information decision module 300 and a correlation map generation module 400, which are respectively described as follows: The image acquisition module 100 is used to acquire multimodal monitoring data of a patient in ear, nose and throat monitoring, and extract an endoscopic monitoring image of the patient from the multimodal monitoring data.

[0023] In specific implementation, first, the patient's ENT multimodal monitoring data can be collected through a variety of channels and equipment. In the present application, the multimodal monitoring data includes endoscopic monitoring images and monitoring text data of the patient in ENT monitoring. The endoscopic monitoring image can be obtained by high-definition electronic nasopharynx and laryngoscope. The endoscopic monitoring image contains anatomical structures such as nasopharynx, throat, and glottis, which are often accompanied by disturbances such as mucus and reflections. The monitoring text data includes the text format of voice and audio data, physiological monitoring data, and structured / unstructured medical record information. In actual implementation, voice and audio data can be collected through a high-sensitivity microphone and an endoscope-attached recording module, physiological monitoring data can be collected through a portable sleep monitor or an intelligent vital sign collection patch, and the patient's structured / unstructured medical record information can be obtained through the hospital HIS system; then, the patient's endoscopic monitoring image can be extracted from the multimodal monitoring data by data traversal and retrieval.

[0024] The image restoration module 200 is used to perform edge monitoring on the endoscopic monitoring image to obtain an endoscopic monitoring edge image, generate a secretion disturbance mask based on the endoscopic monitoring edge image, and determine the patient's endoscopic monitoring restoration image based on the secretion disturbance mask and the endoscopic monitoring image.

[0025] In specific implementation, the Canny edge detection algorithm can be used to perform edge detection on the endoscopic monitoring image, thereby obtaining an endoscopic monitoring edge image; it should be noted that the Canny edge detection algorithm efficiently detects the edge of the endoscopic monitoring image through multi-stage processing (such as noise suppression, gradient calculation, non-maximum suppression, etc.), thereby obtaining an endoscopic monitoring edge image.

[0026] Preferably, in this embodiment, reference Figure 2 As shown in FIG. 1 , this figure is an exemplary flow chart of generating a secretion disturbance mask in an embodiment of the present application. In this embodiment, generating a secretion disturbance mask based on the endoscope monitoring edge image can be specifically implemented by the following steps: In step S21, morphological processing is performed on the endoscope monitoring edge image to obtain a closed edge image; In step S22, the closed edge image is hole-filled to obtain a simply connected region image; In step S23, the simply connected region image is corroded, and a secretion disturbance mask is generated according to the corrosion processing result.

[0027] In the specific implementation, first, the endoscope monitoring edge image can be morphologically processed, that is, the endoscope monitoring edge image can be processed using a morphological closing operation, and the small gaps in the edge can be filled using a dilation operation, and then the original shape of the edge can be restored using an erosion operation. This operation can eliminate the problem of edge breakage or discontinuity, and make the edge of the background area of ​​the image coherent and closed, so that a closed edge image can be obtained, wherein the structural element used in the closing operation can be a circle or an ellipse to adapt to the characteristics of the edge of the background area in the endoscope monitoring image; then, the closed edge image can be hole-filled. It should be noted that the purpose of hole filling is to completely fill the pixels inside the closed area with foreground values, eliminate the holes inside the edge, and thus form a complete and continuous single-connected area. domain, that is, a simply connected region image; further, the simply connected region image can be eroded, that is, a morphological erosion operation is performed on the filled simply connected region image to remove irregular edges or adhesion parts and reduce the boundary of the secretion region. The erosion operation can further eliminate artifacts or abnormal edges in the non-secretion region, making the mask of the secretion region more accurate; finally, a secretion disturbance mask can be generated according to the corrosion processing result, that is, according to the result after the corrosion processing, the eroded region is extracted as the secretion disturbance mask, wherein the secretion disturbance mask is a mask image used to remove the disturbance of the secretion region, and the pixel value corresponding to the secretion disturbance region in the secretion disturbance mask is set to 1, and the pixel values ​​of the remaining regions are set to 0.

[0028] In this embodiment, determining the patient's endoscopic monitoring restoration image based on the secretion disturbance mask and the endoscopic monitoring image is performed by performing an XOR operation on the secretion disturbance mask and the endoscopic monitoring image to obtain the patient's endoscopic monitoring restoration image.

[0029] In the specific implementation, first, the secretion disturbance mask is matched with the original endoscopic monitoring image at the pixel level; then, each pixel in the secretion disturbance mask can be XORed with the corresponding pixel of the endoscopic monitoring image, and its logical rule is that when the pixel value in the secretion disturbance mask is 1 (indicating the secretion disturbance area), the XOR operation is performed with the corresponding pixel of the endoscopic monitoring image, and the result is 0, that is, the disturbance information of the area is removed or cleared; when the mask pixel value is 0 (indicating the non-secretion disturbance area), the XOR operation is performed with the corresponding pixel of the endoscopic monitoring image, and the result retains the original endoscopic monitoring image pixel value; it should be noted that the result of the XOR operation is the patient's endoscopic monitoring restoration image, and the XOR operation of the secretion disturbance mask with the original image can eliminate or weaken the interference factors without destroying the normal structure, thereby improving the image diagnostic value.

[0030] It should be noted that in ENT endoscopic monitoring, the processing flow of edge extraction of endoscopic monitoring images, generation of secretion perturbation masks and construction of endoscopic monitoring restoration images can effectively improve the structural clarity and lesion visibility of the image, significantly suppress image noise and information loss caused by secretions, blur and occlusion, etc., and not only enhance the edge contrast of key anatomical areas (such as the nasal cavity, throat wall, mucosal folds, etc.), but also provide a more stable basic image for subsequent image segmentation and lesion detection.

[0031] The information decision module 300 is used to determine the pixel decision window of each pixel point in the endoscopic monitoring restoration image, and make information decisions on each pixel point in the endoscopic monitoring restoration image based on the corresponding pixel decision window, so as to obtain the patient's endoscopic monitoring decision image.

[0032] Preferably, in this embodiment, reference Figure 3 As shown in FIG. 1 , this figure is an exemplary flow chart of determining the pixel decision window of each pixel point in the endoscopic monitoring restoration image in an embodiment of the present application. In this embodiment, determining the pixel decision window of each pixel point in the endoscopic monitoring restoration image can be specifically implemented by the following steps: In step S31, a preset pixel window is obtained; In step S32, for each pixel point in the endoscope monitoring restoration image, the fineness of information corresponding to the pixel point is determined according to the pixel window; In step S33, the pixel decision window of the pixel point is determined based on the information fineness, and then the pixel decision window of each pixel point in the endoscope monitoring restoration image is obtained.

[0033] In the specific implementation, first, the size of the pixel window can be pre-set based on historical experience and data analysis, which will not be repeated here; then, for each pixel point in the endoscope monitoring and restoration image, the information fineness corresponding to the pixel point can be determined according to the pixel window, and the pixel decision window of the pixel point is determined based on the information fineness, wherein the pixel decision window is a pixel window centered on the corresponding pixel point. The pixel decision window of each pixel point in the endoscope monitoring and restoration image can be obtained in the above manner.

[0034] In this embodiment, the information fineness corresponding to the pixel point is determined according to the pixel window in the following manner, namely: Taking the pixel point as the window center of the pixel window, and then determining all the neighborhood windows of the pixel window; Determining a pixel heterogeneity coefficient between the pixel window and each neighborhood window; The information fineness corresponding to the pixel point is determined by all pixel heterogeneity coefficients.

[0035] In specific implementation, first, the pixel point can be used as the window center of the pixel window, so that multiple neighborhood windows with the same size as the pixel window can be set around the current pixel window; then, the pixel heterogeneity coefficient between the pixel window and each neighborhood window can be determined, wherein the pixel heterogeneity coefficient represents the degree of difference between the pixel included in the pixel window and the pixel included in the neighborhood window. In actual implementation, the mean square error of the pixel values ​​between the pixel point included in the pixel window and the pixel point included in the neighborhood window can be calculated, so that the calculation result is used as the pixel heterogeneity coefficient between the pixel window and the neighborhood window. The pixel heterogeneity coefficient between the pixel window and each neighborhood window can be obtained in the above manner; finally, the information subtlety corresponding to the pixel point can be determined through all the pixel heterogeneity coefficients, wherein the information subtlety represents the information subtlety of the area corresponding to the pixel point, and the information subtlety represents that the more detail information the area corresponding to the pixel point contains, the greater the difference between the area corresponding to the pixel point and other areas. In actual implementation, the mean value of all pixel heterogeneity coefficients can be used as the information subtlety corresponding to the pixel point.

[0036] In this embodiment, the pixel decision window of the pixel point is determined based on the information subtlety in the following manner, namely: Get the preset subtlety threshold; When the information subtlety is greater than the subtlety threshold, setting the pixel decision window size of the pixel point to 3 pixels×3 pixels; When the information subtlety is not greater than the subtlety threshold, the pixel decision window size of the pixel point is set to 7 pixels×7 pixels.

[0037] In specific implementation, first, the subtlety threshold can be pre-set based on experience or experiments according to different endoscopic monitoring image features or specific application scenarios, which will not be repeated here; then, when the information subtlety is greater than the subtlety threshold, it indicates that the information subtlety of the area where the pixel is located is large, which means that the organizational structure of the area is complex and the texture changes are rich. In order to enhance the ability to maintain local features, a smaller 3×3 pixel window is used for information decision processing to ensure accuracy, that is, the pixel decision window size of the pixel point can be set to 3 pixels×3 pixels; when the information subtlety is not greater than the subtlety threshold, it indicates that the area where the pixel is located is relatively uniform and the changes are small. A larger 7×7 pixel window can be used for information decision, which helps to reduce noise interference and improve stability, that is, the pixel decision window size of the pixel point can be set to 7 pixels×7 pixels.

[0038] In this embodiment, information decision is made on each pixel point in the endoscopic monitoring restoration image based on the corresponding pixel decision window, and then the endoscopic monitoring decision image of the patient is obtained in the following manner, namely: Determine the pathological background noise of the restored images of endoscope monitoring; For each pixel point in the endoscope monitoring restoration image, the local information complexity and local pixel level of the pixel point are determined according to the corresponding pixel decision window; Performing information decision on the pixel point through the pathological background noise, the local information complexity and the local pixel level to obtain a decision pixel point corresponding to the pixel point, and then obtaining a decision pixel point corresponding to each pixel point in the endoscope monitoring restoration image; The patient's endoscopic monitoring decision image is constructed based on all decision pixels.

[0039] In the specific implementation, first, the pathological background noise of the endoscopic monitoring restoration image can be determined, wherein the pathological background noise represents the background noise level in the endoscopic monitoring restoration image, and the background noise variance of the endoscopic monitoring restoration image can be calculated, thereby using the calculation result as the pathological background noise of the endoscopic monitoring restoration image; then, for each pixel point in the endoscopic monitoring restoration image, the local information complexity and local pixel level of the pixel point can be determined according to the corresponding pixel decision window, wherein the local information complexity represents the complexity of the pixel information contained in the pixel decision window corresponding to the pixel point, the variance of all pixels contained in the pixel decision window corresponding to the pixel point can be used as the local information complexity of the pixel point, and the local pixel level represents the mean value of the pixel value contained in the pixel decision window corresponding to the pixel point.

[0040] In addition, in specific implementation, information decision can be made on the pixel points through pathological background noise, local information complexity and local pixel level, so as to obtain the decision pixel point corresponding to the pixel point. In actual implementation, the pixel value of the decision pixel point corresponding to the pixel point can be determined by the following formula:

[0041] in, Represents the pixel value of the decision pixel corresponding to pixel i, represents the pixel value of pixel i, represents the local information complexity of pixel i, represents the local pixel level of pixel i, Represents the pathological background noise. The decision pixel points corresponding to each pixel point in the endoscopic monitoring restoration image can be obtained by the above method. Finally, the patient's endoscopic monitoring decision image can be constructed based on all the decision pixel points, that is, all the decision pixel points are combined according to the corresponding image coordinates to obtain the patient's endoscopic monitoring decision image.

[0042] It should be noted that in the endoscopic monitoring restoration image, the pixel decision window is adaptively determined for each pixel point, and local information decision is made based on it, which helps to fine-tune the image processing strategy: using a small window in detail-rich areas to retain edges and lesion contours, and using a large window in smooth areas to suppress noise and artifacts, thereby significantly improving the overall image quality and structural clarity. By enhancing the edge discernibility and regional consistency of the image, it can effectively assist subsequent image semantic segmentation, abnormality recognition and tissue structure determination.

[0043] The association map generation module 400 is used to extract the patient's monitoring text data from the multimodal monitoring data, perform feature fusion on the monitoring text data and the endoscopic monitoring decision image, and then generate a fused association map of the patient in ENT monitoring.

[0044] In specific implementation, the patient's monitoring text data can be extracted from the multimodal monitoring data by traversal and retrieval.

[0045] In this embodiment, the monitoring text data and the endoscope monitoring decision image are feature fused to generate a fusion association map of the patient in ENT monitoring, which can be specifically implemented in the following manner, namely: Performing semantic analysis on the monitoring text data to obtain structured knowledge of the patient in ENT monitoring; Extracting lesion features from the endoscope monitoring decision image to obtain a lesion image feature vector; The structured knowledge and the lesion image feature vector are jointly modeled to generate a fused association map of the patient in ENT monitoring.

[0046] In specific implementation, first, the monitoring text data can be semantically parsed, that is, the monitoring text data can be processed using natural language processing technology, including word segmentation, named entity recognition (NER), relationship extraction and other steps. These technologies are used to extract key information in the text, thereby converting the extracted text information into structured data, that is, the patient's structured knowledge in ENT monitoring. For example, the ENT-related examination records, doctor's notes, postoperative summaries and other text data are pre-processed, including removing noise characters, unifying medical terms, using the pre-trained language model in the medical field, combined with a customized ENT disease entity dictionary, to identify key entities in the text, such as lesion name, anatomical site, lesion nature and time status, etc., and using dependency syntactic analysis, attention mechanism or relation extraction model to identify the semantic relationship between entities, forming structured knowledge triples as the patient's semantic knowledge representation in ENT monitoring; then, the lesion feature extraction can be performed on the endoscopic monitoring decision image, that is, the endoscopic monitoring decision image is extracted through a deep learning model (such as a convolutional neural network CNN). Automatic identification and segmentation of inflammatory patches, secretion-covered areas, cyst contours, etc., so as to extract various features of the endoscopic monitoring decision-making image area, such as texture features (grayscale co-occurrence matrix, local binary pattern, etc.), morphological features (region size, shape, boundary, etc.), color features, edge features, etc., and the vector composed of all extracted features is used as the lesion image feature vector; finally, the structured knowledge and the lesion image feature vector can be jointly modeled, that is, the structured knowledge and the lesion image feature vector can be fused by the joint modeling technology to form a joint representation of multimodal data, such as the multimodal neural network in deep learning (for example, the cross-modal learning model based on Transformer) takes into account the structured knowledge and the lesion image feature vector information at the same time. On the basis of the joint modeling, the graph construction technology is used to transform the structured knowledge and the lesion image feature vector into nodes in the knowledge graph, and the relationship between the nodes is described as an edge. In the above way, the structured knowledge and the lesion image feature vector can be linked to form a fusion association graph of patients in ENT monitoring.

[0047] It should be noted that the feature fusion of monitoring text data and endoscopic monitoring decision images to generate a fusion association map of patients in ENT monitoring has significant clinical and intelligent analysis value. By fusing the structured semantic information in the text with the characteristics of the lesion area in the image, multi-modal information complementarity and semantic enhancement can be achieved, which can improve the image comprehension ability and semantic clarity. Through image decision optimization processing, interference areas such as secretions are removed to improve the overall image quality. The fusion association map generated by fusion constructs a patient's individualized knowledge network, which is convenient for time series analysis, pathological evolution reasoning and intelligent auxiliary diagnosis.

[0048] By extracting the patient's monitoring text data from the pathological information and combining it with the endoscopic monitoring decision image to generate the patient's fusion association map, the structured text data can be deeply integrated with the image features to provide comprehensive patient pathology information. By associating the image features of the lesion area with the clinical diagnosis and staging information in the report, the image interpretation accuracy and visualization effect can be enhanced, and the automatic integration of pathological knowledge can be achieved.

[0049] It can be seen that in this application, first, in ENT endoscope monitoring, edge extraction is performed on the endoscopic monitoring image, secretion disturbance mask is generated, and a processing flow of the endoscopic monitoring restoration image is constructed, which can effectively improve the structural clarity and lesion visibility of the image, and significantly suppress image noise and information loss caused by secretions, blur occlusion, etc.; then, a pixel decision window is adaptively determined for each pixel in the endoscopic monitoring restoration image, and local information decisions are made accordingly, that is, a small window is used in detail-rich areas to retain edges and lesion contours, and a large window is used in smooth areas to suppress noise and artifacts, thereby significantly improving the overall image quality and structural clarity; finally, the monitoring text data is feature fused with the endoscopic monitoring decision image to generate a fused association map of the patient in ENT monitoring. By fusing the structured semantic information in the text with the lesion area features in the image, information complementarity and semantic enhancement between multiple modalities can be achieved, which can improve the image comprehension ability and semantic clarity, remove interference areas such as secretions through image decision optimization processing, improve the overall image quality, and realize knowledge fusion in ENT monitoring.

[0050] In summary, the technical solution adopted in this application can improve the image quality of endoscopic monitoring images, and generate a patient association map in combination with the monitoring text to achieve knowledge fusion in ENT monitoring.

[0051] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0052] A person skilled in the art may understand that all or part of the steps in the various methods of the above embodiments may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, the storage medium including a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically-erasable programmable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0053] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

Claims

1. An intelligent otolaryngology diagnosis and monitoring system based on multimodal data fusion, characterized in that: The diagnostic and monitoring system comprises: An image acquisition module, used to acquire multimodal monitoring data of the patient in ENT monitoring, and extract the patient's endoscopic monitoring image from the multimodal monitoring data; an image restoration module, configured to perform edge monitoring on the endoscopic monitoring image to obtain an endoscopic monitoring edge image, generate a secretion disturbance mask based on the endoscopic monitoring edge image, and determine an endoscopic monitoring restoration image of the patient based on the secretion disturbance mask and the endoscopic monitoring image; An information decision module is used to determine a pixel decision window for each pixel point in the endoscopic monitoring restoration image, and to make information decisions on each pixel point in the endoscopic monitoring restoration image based on the corresponding pixel decision window, thereby obtaining an endoscopic monitoring decision image of the patient; The association map generation module is used to extract the patient's monitoring text data from the multimodal monitoring data, perform feature fusion on the monitoring text data and the endoscopic monitoring decision image, and then generate a fused association map of the patient in ENT monitoring.

2. The intelligent otolaryngology diagnosis and monitoring system based on multimodal data fusion according to claim 1, characterized in that: The multimodal monitoring data includes endoscopic monitoring images and monitoring text data of patients in ear, nose and throat monitoring.

3. The intelligent otolaryngology diagnosis and monitoring system based on multimodal data fusion according to claim 1, characterized in that: The Canny edge detection algorithm is used to perform edge detection on the endoscope monitoring image.

4. The intelligent otolaryngology diagnosis and monitoring system based on multimodal data fusion according to claim 1, characterized in that: Generating a secretion disturbance mask according to the endoscope monitoring edge image specifically includes: Performing morphological processing on the endoscope monitoring edge image to obtain a closed edge image; Filling holes in the closed edge image to obtain a simply connected region image; The singly connected region image is corroded, and a secretion disturbance mask is generated according to the corrosion processing result.

5. The intelligent otolaryngology diagnosis and monitoring system based on multimodal data fusion according to claim 1, characterized in that: Determining the patient's endoscopic monitoring restoration image based on the secretion disturbance mask and the endoscopic monitoring image is to perform an XOR operation on the secretion disturbance mask and the endoscopic monitoring image, thereby obtaining the patient's endoscopic monitoring restoration image.

6. The intelligent otolaryngology diagnosis and monitoring system based on multimodal data fusion according to claim 1, characterized in that: The pixel decision window for determining each pixel point in the endoscope monitoring restoration image specifically includes: Get the preset pixel window; For each pixel point in the endoscope monitoring restoration image, determining the information fineness corresponding to the pixel point according to the pixel window; The pixel decision window of the pixel point is determined based on the information subtlety, and then the pixel decision window of each pixel point in the endoscope monitoring restoration image is obtained.

7. The intelligent otolaryngology diagnosis and monitoring system based on multimodal data fusion according to claim 6, characterized in that: Determining the information fineness corresponding to the pixel point according to the pixel window specifically includes: Taking the pixel point as the window center of the pixel window, and then determining all the neighborhood windows of the pixel window; Determining a pixel heterogeneity coefficient between the pixel window and each neighborhood window; The information fineness corresponding to the pixel point is determined by all pixel heterogeneity coefficients.

8. The intelligent otolaryngology diagnosis and monitoring system based on multimodal data fusion according to claim 6, characterized in that: Determining the pixel decision window of the pixel point based on the information subtlety specifically includes: Get the preset subtlety threshold; When the information subtlety is greater than the subtlety threshold, setting the pixel decision window size of the pixel point to 3 pixels×3 pixels; When the information subtlety is not greater than the subtlety threshold, the pixel decision window size of the pixel point is set to 7 pixels×7 pixels.

9. The intelligent otolaryngology diagnosis and monitoring system based on multimodal data fusion according to claim 1, characterized in that: Based on the corresponding pixel decision window, information decision is made on each pixel point in the endoscopic monitoring restoration image, and then the endoscopic monitoring decision image of the patient is obtained, which specifically includes: Determine the background noise floor of the endoscope monitoring restored image; For each pixel point in the endoscope monitoring restoration image, the local information complexity and local pixel level of the pixel point are determined according to the corresponding pixel decision window; Performing information decision on the pixel point through the background noise, the local information complexity and the local pixel level to obtain a decision pixel point corresponding to the pixel point, and then obtaining a decision pixel point corresponding to each pixel point in the endoscope monitoring restoration image; The patient's endoscopic monitoring decision image is constructed based on all decision pixels.

10. The intelligent otolaryngology diagnosis and monitoring system based on multimodal data fusion according to claim 1, characterized in that: The feature fusion of the monitoring text data and the endoscope monitoring decision image to generate a fusion association map of the patient in ENT monitoring specifically includes: Performing semantic analysis on the monitoring text data to obtain structured knowledge of the patient in ENT monitoring; Extracting lesion features from the endoscope monitoring decision image to obtain a lesion image feature vector; The structured knowledge and the lesion image feature vector are jointly modeled to generate a fused association map of the patient in ENT monitoring.

Images