Multi-modal data fused medical and educational informatization comprehensive supervision and diagnosis platform system

Abstract

The invention discloses a medical and educational informatization comprehensive supervision and diagnosis platform system based on multi-modal data fusion. A data acquisition module acquires multi-modal data from a medical and educational information system and preprocesses the multi-modal data; the data storage module constructs a hybrid architecture to classify and store data and establishes an index. The multi-modal data fusion module deeply fuses medical records, images and physiological monitoring data of medical treatment and teaching resources, learning behaviors and teaching evaluation data features of education by means of a feature extraction technology and a fusion algorithm. The supervision and diagnosis module constructs models of medical service quality evaluation, disease diagnosis assistance, teaching quality monitoring and the like based on fusion features to realize accurate analysis. The user interaction module provides interaction interfaces for different roles and supports operations such as data query and uploading. The platform system breaks the data barrier, fully mines the data value, provides powerful support for informatization management, decision making and service optimization in the medical and education fields, and promotes the innovative development of the two fields.

Description

Technical Field

[0001] The present invention relates to the field of information technology and data processing, and in particular focuses on building an innovative multimodal data fusion platform system, which aims to provide strong technical support for the comprehensive information-based supervision and precise diagnosis in the two key areas of medical care and education, so as to solve many problems such as data isolation and one-sided analysis in the existing medical and education information systems, and promote efficient management and quality service improvement in these two fields. Background Art

[0002] As the medical industry is booming, the wave of digitalization has led to an explosive growth in various types of medical data. Clinical medical records contain rich text information, covering the patient's basic situation, medical history, symptom description, diagnosis results, and treatment process; medical imaging data such as X-rays, CT scans, and MRI images intuitively present the internal structure and pathological characteristics of the human body; in addition, there are real-time data from various physiological monitoring devices, such as heart rate, blood pressure, and blood oxygen saturation. However, these data are often stored in different information systems and each operates independently. The hospital information management system (HIS) focuses on administrative affairs and basic medical information management, the medical image storage and transmission system (PACS) mainly processes imaging data, and physiological monitoring data may be scattered in a dedicated monitoring device database. This data dispersion leads to huge challenges in comprehensively analyzing the patient's condition, evaluating the quality of medical services, and optimizing the allocation of medical resources. For example, when making a diagnosis, doctors may need to switch between multiple systems to query data, making it difficult to quickly obtain a patient's panoramic information, and the potential correlation information between different modal data has not been fully mined and utilized, limiting the accuracy and efficiency of disease diagnosis.

[0003] The education sector is also facing the dilemma of data management and analysis. With the acceleration of the process of educational informatization, teaching resource data is becoming increasingly rich and diverse, including electronic textbooks, teaching courseware, online course videos, etc.; student learning process data such as learning time records, online learning behaviors (clicks, browsing, asking questions, answering questions, etc.), homework and test scores are accumulated in large quantities; teaching evaluation data comes from multiple aspects such as teacher evaluation, student mutual evaluation and self-evaluation. However, most current educational information systems are designed in a functional module segmentation style. For example, the teaching resource management platform only focuses on the storage and distribution of resources, and the student performance management system focuses on performance statistics and analysis, and fails to achieve deep integration and collaborative analysis of multimodal data. This makes it difficult for educational decision makers to fully understand the actual teaching effect, accurately grasp students' learning needs and status, and cannot provide a scientific basis for the optimization of teaching strategies, personalized learning support, and the rational allocation of educational resources, hindering the overall improvement of education quality and the effective promotion of educational equity.

[0004] To sum up, there is an urgent need for an innovative technical solution that can break down the data barriers in the medical and educational fields, organically integrate multimodal data, and build an intelligent comprehensive supervision and diagnosis platform system to maximize the mining and utilization of data value and promote the medical and educational industries to new heights of development. Summary of the invention

[0005] In order to overcome the above problems or at least partially solve the above problems, the core of the present invention is to provide a medical and educational information comprehensive supervision and diagnosis platform system with multimodal data fusion. The system realizes the efficient collection, precise fusion, in-depth analysis, intelligent supervision and diagnosis of massive multimodal data in the medical and educational fields through a series of innovatively designed functional modules and advanced data processing technologies, thereby providing comprehensive and accurate decision-making support for improving the quality of medical services, precise treatment of diseases, optimization of education and teaching, and personalized development of students.

[0006] In the first aspect, the multimodal data fusion medical and educational information-based integrated supervision and diagnosis platform system of the present invention is mainly composed of the following key modules:

[0007] Data acquisition module:

[0008] Medical data acquisition submodule: A multi-source data acquisition interface is designed specifically for medical information systems. Connect with the HIS system to obtain structured information such as basic patient information and medical record text data, and use specific data extraction protocols to ensure data integrity and accuracy. For the PACS system, use image acquisition technology to obtain various types of medical imaging data in accordance with a predetermined image format (such as the DICOM standard) and transmit them in real time. At the same time, establish a wireless or wired connection with physiological monitoring equipment to receive real-time physiological monitoring data from devices such as electrocardiograms (ECGs), electroencephalograms (EEGs), and blood pressure monitors, and ensure data reliability and continuity through data verification algorithms.

[0009] Educational data collection submodule: In order to adapt to the diversity of educational information systems, a variety of data collection channels are developed. Connect with the online learning platform and obtain students' learning behavior data through the API interface, including learning time distribution, order and frequency of learning resource access, and interactive operation records (such as forum posts, likes, comments, etc.). Extract structured information such as students' course schedules, grade data, and teacher evaluation data from the school's academic affairs management system. For teaching resource data, such as teaching videos, courseware documents, etc., web crawler technology is used in combination with resource library access rights verification to ensure that metadata (such as resource type, subject, creation time, etc.) and resource content data of various teaching resources are collected legally and compliantly, and are classified and sorted. The collected medical and educational data are subject to a strict data cleaning process to remove duplicate data, erroneous data, and incomplete data. At the same time, the data is converted to a standardized format, such as unifying the encoding format of text data and adjusting the image data to a unified resolution, etc., laying a solid foundation for subsequent data processing.

[0010] Data storage module: Build a hybrid data storage architecture that integrates the advantages of relational and non-relational databases. For structured data in the medical and educational fields, such as patient basic information tables, student score tables, and course information tables, relational databases (such as MySQL, Oracle, etc.) are used for storage, and their perfect transaction processing mechanism and powerful structured query language (SQL) functions are used to ensure data consistency, integrity, and efficient query. For unstructured data such as medical images, teaching videos, and audio, non-relational databases (such as MongoDB, HBase, etc.) or distributed file systems (such as HDFS) are used for storage. During the storage process, multi-dimensional indexes are established based on key attributes such as the data's modality, source, and timestamp. For example, for medical imaging data, indexes are established according to patient ID, imaging type, and examination time, so that the required data can be quickly located and retrieved in massive data. At the same time, data partitioning technology is used to store data from different time periods or different medical institutions and schools in different storage areas, further improving data query and management efficiency.

[0011] Multimodal data fusion module: Feature extraction submodule: For medical data of different modalities, a variety of advanced technologies are used for feature extraction. For medical image data, the convolutional neural network (CNN) architecture in deep learning is used to automatically learn the texture, shape, edge and other features in the image through the combination of convolutional layer, pooling layer and fully connected layer, and convert the image data into a low-dimensional and representative feature vector. For example, in the analysis of lung CT images, CNN can accurately extract the size, shape, density and other feature information of lung nodules. For medical record text data, natural language processing (NLP) technology is used, including word vector models (such as Word2Vec, GloVe, etc.) to map the words in the text into vector representations, and then recurrent neural networks (RNN) or their variants (such as long short-term memory networks (LSTM), gated recurrent units (GRU)) are used to perform sequence modeling on the medical record text to extract the semantic features of key information such as symptom descriptions, disease development process, and treatment plans. In the field of education, data mining algorithms are used to extract features from student learning behavior data. For example, students' learning behavior patterns are classified through cluster analysis to extract the behavioral characteristics of students with different learning styles (such as visual, auditory, and kinesthetic). For teaching resource data, text analysis technology is used to extract resource subject keywords, knowledge difficulty levels and other features.

[0012] Fusion algorithm submodule: Use a multimodal fusion model based on deep learning, such as a multimodal attention mechanism fusion network. This model can automatically assign weights according to the importance of different modal features, and deeply fuse feature vectors of different modalities in the medical and educational fields. For example, in a medical diagnosis scenario, when medical image features are fused with medical record text features, the model will dynamically adjust the weight ratio of image features and text features in the fusion process according to the specific disease type and diagnostic needs. If the image feature has a higher diagnostic value for a certain disease, the image feature is given a greater weight, and vice versa, the text feature is emphasized. In this way, a highly representative fusion feature vector containing multimodal information is generated, fully exploring the complementarity and correlation between multimodal data, and providing richer and more accurate information input for subsequent supervision and diagnosis.

[0013] Supervision and diagnosis module: Medical supervision and diagnosis submodule: Based on the fused multimodal medical data feature vector, a multi-dimensional medical supervision and diagnosis model is constructed. A large amount of historical medical data is trained using machine learning algorithms (such as support vector machine (SVM), decision tree, random forest, etc.) to establish a medical service quality assessment model, which can comprehensively evaluate the quality of medical services from multiple aspects such as patient medical treatment process, treatment effect, and medical resource consumption. For example, by analyzing the patient's waiting time for treatment, length of hospital stay, treatment costs, cure rate, recurrence rate and other data, the service efficiency and quality level of the medical institution can be judged. A disease diagnosis auxiliary model is constructed, and deep learning algorithms (such as deep belief network (DBN), generative adversarial network (GAN), etc.) are used to explore the potential relationship between diseases and multimodal data features, and provide doctors with auxiliary suggestions for disease diagnosis, such as predicting possible types of diseases based on the patient's symptom text description, imaging examination results, and physiological monitoring data, and providing treatment plan references for similar cases. At the same time, a medical resource allocation optimization model is established. By analyzing data such as patient flow, disease distribution, and medical resource utilization in different regions and medical institutions, and using optimization algorithms such as linear programming and integer programming, the rational allocation and dynamic adjustment of medical resources (such as beds, medical staff, medical equipment, etc.) can be achieved, thereby improving the overall utilization efficiency of medical resources.

[0014] Education supervision and diagnosis submodule: Based on the fused educational multimodal data feature vector, an education information supervision and diagnosis model is constructed. A teaching quality monitoring model is constructed using data mining and analysis technology to comprehensively monitor and evaluate teaching quality from multiple dimensions, including teacher teaching behavior (such as the diversity of teaching methods, frequency of classroom interaction, timeliness of homework correction, etc.), student learning effects (such as the extent of grade improvement, knowledge mastery, changes in learning attitudes, etc.), and teaching resource utilization (such as teaching resource click-through rate, resource update frequency, etc.). A student learning status analysis model is constructed, and machine learning algorithms (such as cluster analysis, association rule mining, etc.) are used to conduct in-depth analysis of students' learning behavior characteristics, identify students' learning difficulties, learning interest preferences, and learning style characteristics, and provide students with personalized learning suggestions and learning path planning. For example, if it is found that students spend too much time studying on a specific knowledge point in a subject and have a high error rate in answering questions, it is judged as a learning difficulty, and targeted tutoring resources or learning strategies are recommended. Establish a teaching strategy adjustment suggestion model. Based on the results of teaching quality monitoring and student learning status analysis, use expert systems or intelligent recommendation algorithms to provide teachers with teaching strategy adjustment suggestions, such as optimization directions for teaching content, improvement measures for teaching methods, and reasonable adjustments to teaching progress, so as to improve teaching effectiveness and student learning experience.

[0015] User interaction module: Design a multi-role, multi-level user interaction interface. For medical personnel (such as doctors, nurses, hospital managers, etc.), a patient data query interface is provided, which can quickly query the patient's detailed medical data according to the patient ID, name, disease type and other methods, including fused multimodal data display, supervision and diagnosis result report, etc. Doctors can call the platform system in real time during the diagnosis process to obtain auxiliary information for disease diagnosis, and feedback the diagnosis results and treatment plans to the platform to achieve closed-loop management of data. For hospital managers, functional interfaces such as medical service quality statistical reports and medical resource utilization visualization charts are provided to facilitate macro-management decisions and resource allocation. In the field of education, teachers are provided with a comprehensive analysis interface for the learning status of class students, which can view the learning status, grade distribution, learning behavior characteristics and other information of individual or group students, receive suggestions for adjusting teaching strategies, and upload teaching practice data and teaching reflection records to enrich platform data resources. Design a personalized learning space for students to display learning progress, analysis of learning advantages and disadvantages, personalized learning resource recommendations and other content. Students can interact with the platform according to their own needs, such as feedback on learning experience and selection of learning resources of interest. Provide education managers with an overall overview of the school or region's education, including teaching quality assessment results, distribution and utilization of educational resources, etc., so that they can formulate education policies and plan education development strategies. In addition, the user interface also has a system setting function. Users can set parameters such as data display format, query conditions, and warning thresholds according to their own needs. It also supports data upload functions, such as medical personnel can upload special case data, and teachers can upload self-made teaching resources, etc., to further improve the data resource library of the platform system and enhance the adaptability and practicality of the system.

[0016] In the second aspect, the implementation method of the embodiment of the present application provides a multi-modal data fusion medical and education information comprehensive supervision and diagnosis platform system, and the specific steps are as follows:

[0017] Step 1: Data collection and preprocessing

[0018] Medical data acquisition and preprocessing: The data acquisition submodule establishes a secure connection with the HIS system, extracts basic patient information (such as name, gender, age, contact information, etc.) and medical record text data (including admission records, medical records, discharge summaries, etc.) from the database according to the predetermined SQL query statement, and converts them into a unified JSON format for subsequent transmission and processing. The connection with the PACS system follows the DICOM protocol, obtains medical image data in real time, and stores them in a temporary buffer area according to the image type (such as X-ray, CT, MRI, etc.). At the same time, it connects to the physiological monitoring equipment through Bluetooth or wired network, receives physiological monitoring data, and performs data integrity verification, such as checking the header and tail identifiers of the data packet, data verification, etc. In the data preprocessing stage, the collected medical data is cleaned. Remove redundant information in the medical record text (such as repeated symptom descriptions, meaningless punctuation, etc.) and correct erroneous data (such as incorrect age records, typos, etc.). For medical image data, image enhancement processing is performed, such as adjusting contrast and brightness, to improve image quality for subsequent feature extraction. The physiological monitoring data is smoothed to remove noise interference, and the data collected by different devices are synchronized in time to ensure the consistency of the data in the time dimension.

[0019] Educational data collection and preprocessing: The educational data collection submodule interacts with the online learning platform through the API interface to obtain students' learning behavior data. For example, by calling the platform's log recording interface, the time when students log in to the platform each time, the name of the course, the length of study, the pause, fast forward, and rewind operation records in the course video, and the posting and reply content in the course discussion area are obtained, and organized into a structured data table. Extract students' course schedule information (such as course name, lecturer, class time, classroom location, etc.), grade data (including regular homework grades, test scores, comprehensive grades, etc.) and teacher evaluation data (such as teaching attitude, teaching methods, teaching effectiveness and other evaluation index scores) from the academic affairs management system, and use data encryption transmission technology to ensure data security. For teaching resource data, use web crawler technology to collect from legitimate educational resource websites or school internal resource libraries. For example, for teaching video resources, not only the video file itself is collected, but also the metadata of the video, such as video title, introduction, upload time, playback time, subject, etc.; for teaching courseware documents, extract the title, keywords, chapter structure and other contents of the document. In the pre-processing phase, the educational data is cleaned and organized. Invalid operation records in the learning behavior data (such as abnormal clicks caused by network failures, etc.) are removed, and outlier detection and correction are performed on the performance data (such as data that obviously deviates from the normal performance range). The teaching resource data is classified and organized, and a resource directory is established according to attributes such as subject, grade, and resource type to facilitate subsequent management and query.

[0020] Step 2: Data storage

[0021] Preprocessed medical structured data (such as patient basic information table, medical record text structured data, etc.) are stored in a relational database according to the predetermined database table structure. For example, a patient information table is created, which contains fields such as patient ID, name, gender, age, and admission time; a medical record table is created, which contains fields such as medical record ID, patient ID, symptom description, diagnosis results, and treatment plans. Unstructured data such as medical imaging data, physiological monitoring data, and teaching videos and audio in the field of education are stored in non-relational databases or distributed file systems. Taking HDFS as an example, medical imaging data is stored in different folders according to patient ID and image type, and each image file is stored with its unique file name (such as patient ID_image type_examination time.dcm). During the storage process, multi-dimensional indexes are established for various types of data. In the relational database, a primary key index is established for the patient ID field of the patient information table, and a joint index is established for the patient ID and diagnosis result fields of the medical record table, so as to quickly query the medical record information of a specific patient or filter patients according to the diagnosis results. For unstructured data, we can establish metadata-based indexes in the distributed file system. For example, for teaching video resources, we can establish indexes based on metadata such as subject, grade, and keywords, so that we can quickly locate relevant teaching video resources through keyword search. At the same time, we can use data partitioning technology, such as time partitioning to store medical data by year or month, and education data by semester, to improve data query and management efficiency.

[0022] Step 3: Multimodal Data Fusion

[0023] Feature extraction:

[0024] Feature extraction in the medical field: Medical image data is read from the data storage module and input into the pre-trained CNN model. For example, for the CNN model for lung disease diagnosis, its convolution layer can automatically learn the texture features in the lung image, the pooling layer performs dimensionality reduction on the features, and the fully connected layer converts the extracted features into fixed-length feature vectors. For medical record text data, the word vector model is first used to convert the words in the text into vector representations, and then input into the LSTM network. The LSTM network can effectively capture the long-distance semantic dependencies in the text through its special gating structure, and extract key semantic features such as the disease development process and treatment effect feedback. The extracted medical image feature vector is preliminarily combined with the medical record text feature vector to form a multimodal feature representation of medical data.

[0025] Feature extraction in the field of education: Read student learning behavior data, use clustering algorithms (such as K-Means clustering) to classify students' learning behavior patterns, and extract learning behavior features represented by different cluster centers, such as learning behavior features with long learning time and frequent interactions, learning behavior features with short learning time but high homework completion efficiency, etc. For teaching resource data, use text analysis technology (such as TF-IDF algorithm) to extract the theme keyword features of resources. For example, the keywords of a mathematics teaching courseware may include "function", "derivative", "limit", etc., and integrate these feature vectors with student learning behavior feature vectors to form a multimodal feature set of educational data.

[0026] Fusion algorithm: A multimodal fusion network based on the attention mechanism is used, and the multimodal feature representation of medical data and the multimodal feature set of educational data are used as input. The network automatically focuses on the feature information that is more critical to the current supervision and diagnosis tasks by calculating the attention weights between different modal features, thereby achieving a deep fusion of medical and educational multimodal features. For example, when assisting in disease diagnosis and involving specific educational background factors (such as the patient's occupation and learning experience may affect the cause of the disease), the network will give a certain weight to the relevant educational features, and work together with the medical features to generate a fusion feature vector, fully exploring the potential correlation and synergistic value between cross-domain data.

[0027] Step 4: Monitoring and diagnosis

[0028] Medical supervision and diagnosis submodule:

[0029] Medical service quality assessment model: Based on the fusion feature vector, the support vector machine (SVM) algorithm is used to train a large amount of historical medical data. The model comprehensively considers various data in the entire medical process of patients from registration to discharge, such as waiting time, examination and testing process duration, treatment cost details, medical staff response timeliness, etc., as well as the final treatment effect indicators (such as cure rate, improvement rate, recurrence rate, etc.). The hyperplane constructed by SVM classifies medical cases with different service quality levels, thereby evaluating the quality and grading the new medical service data, providing intuitive service quality quantification results and improvement direction suggestions for medical institution management departments.

[0030] Disease diagnosis auxiliary model: A disease diagnosis auxiliary model is constructed using a deep belief network (DBN). DBN is composed of multiple stacked restricted Boltzmann machines (RBMs) and can automatically learn complex feature representations in data. The fused feature vector is input into the DBN. The model mines the deep relationship between disease symptoms, imaging features, physiological monitoring data, and related educational background factors (such as occupational disease tendencies that may be caused by long-term employment in a specific occupation) through unsupervised pre-training and supervised fine-tuning. For example, in the diagnosis of cardiovascular diseases, the model can not only analyze the patient's electrocardiogram, cardiac ultrasound images and other medical data, but also combine factors such as the patient's work stress level (inferred from education and career experience data) to improve the accuracy and comprehensiveness of disease diagnosis, and provide doctors with possible disease type predictions, similar case references, and personalized diagnostic recommendation rankings.

[0031] Medical resource allocation optimization model: The integer programming algorithm is used to construct a medical resource allocation optimization model. The model takes the patient flow data (including outpatient volume and hospitalization volume) of each department of the hospital, the distribution data of disease types, the current status data of different medical resources (such as beds, the number of medical staff, the number of various medical equipment and the frequency of use) and the relevant information in the fusion feature vector as input. Through the integer programming algorithm, medical resources are reasonably allocated and dynamically adjusted on the premise of meeting the medical service quality standards (such as the upper limit of patient waiting time, the lower limit of treatment effect, etc.) and resource constraints (such as the working time limit of medical staff, the equipment maintenance cycle, etc.). For example, according to the high incidence of diseases in different seasons and the distribution of medical resources in the surrounding areas, the internal bed allocation plan of the hospital is optimized, medical staff are deployed to departments with greater demand, or the procurement and update plan of medical equipment is planned to improve the overall utilization efficiency and benefits of medical resources.

[0032] Education supervision and diagnosis submodule:

[0033] Teaching quality monitoring model: A teaching quality monitoring model is constructed by combining principal component analysis (PCA) with linear regression. PCA first reduces the dimensionality of the multi-dimensional data in the fusion feature vector (such as teacher teaching behavior characteristics, student learning effect characteristics, teaching resource utilization characteristics, etc.) and extracts the main components. Then, a relationship model between teaching quality indicators (such as the improvement of students' average grades, students' teaching satisfaction survey scores, etc.) and the main components is established through linear regression. Through this model, various data changes in the teaching process can be monitored in real time, and fluctuation factors of teaching quality can be discovered in time, such as the impact of sudden changes in teachers' teaching methods on students' learning effects, or the decline in students' learning interest due to untimely updates of teaching resources, etc., and targeted improvement suggestions and early warning information can be provided to the teaching management department.

[0034] Student learning status analysis model: A student learning status analysis model is constructed using hierarchical clustering and association rule mining algorithms. First, the hierarchical clustering algorithm divides the student group into different levels of learning status categories based on the student's learning behavior characteristics (such as learning time distribution, learning resource preferences, changes in answer accuracy, etc.), such as excellent and stable learning type, progressive learning type, and learning difficulty type. Then, the association rule mining algorithm mines the correlation between student learning behavior and learning effect within each category, such as discovering the association rules between "long homework completion time and high error rate" and "low class participation and difficulty in understanding specific knowledge points" that often appear in students with learning difficulties. Based on these analysis results, a personalized learning status report is generated for each student, including learning strengths and weaknesses analysis, learning risk warning, and personalized learning suggestions, such as recommending suitable learning resources and guidance on learning method improvement, to help students better understand their learning situation and adjust their learning strategies.

[0035] Teaching strategy adjustment suggestion model: A teaching strategy adjustment suggestion model is constructed based on expert system and collaborative filtering algorithm. The expert system has built-in teaching experience and knowledge rules of experts in the field of education. According to the results of the teaching quality monitoring model and the student learning status analysis model, combined with the current teaching environment and educational goals, it generates preliminary teaching strategy adjustment suggestions, such as the key adjustment direction of teaching content and optimization suggestions for teaching methods. The collaborative filtering algorithm further considers the successful teaching experience of other similar teachers or classes, and supplements and optimizes the preliminary suggestions by analyzing the effective teaching strategies they adopt in similar teaching situations. For example, if students in a class generally have learning difficulties in a certain knowledge point in the subject of mathematics, and the expert system recommends strengthening practice and tutoring, the collaborative filtering algorithm may recommend interactive teaching games or group cooperative learning methods used by other classes when teaching this knowledge point, providing teachers with more reference value and innovative teaching strategy adjustment plans to improve teaching effectiveness and student learning experience.

[0036] Step 5: User Interaction

[0037] Medical staff interaction interface: Design a comprehensive patient data query and diagnosis assistance interface for doctors. Doctors can quickly retrieve the patient's complete medical data by entering key information such as patient ID or name, including medical record text details, medical image display, physiological monitoring data curves, and fused multimodal data feature analysis results. During the diagnosis process, doctors can call the prediction results, similar case reference materials, and personalized diagnosis suggestions provided by the disease diagnosis assistance model at any time, and record their own diagnostic ideas and treatment plans with the platform. The interface also provides a medical service quality assessment report viewing function, so that doctors can understand the overall service level of their department or medical institution and their own service quality ranking, so as to improve the service process and quality in a targeted manner. For nurses, the interactive interface focuses on patient care process management and nursing quality monitoring data display, such as patient care plan arrangement, nursing operation record query, patient rehabilitation progress tracking, etc. At the same time, they can also view medical resource allocation information related to nursing work, such as ward bed usage, nursing equipment deployment information, etc., to ensure the efficiency and safety of nursing work.

[0038] Educator Interactive Interface: Create a panoramic analysis of the learning situation of students in the class and a teaching strategy support interface for teachers. Teachers can view individual learning status reports of students in the class they teach, including detailed information such as learning progress, knowledge mastery distribution, and learning behavior characteristics analysis, so as to timely discover students' learning problems and provide personalized tutoring. The teaching quality evaluation report generated by the teaching quality monitoring model is displayed in the interface in the form of intuitive charts, such as teaching effect trend charts, student score distribution histograms, etc., to help teachers understand the advantages and disadvantages of their teaching work. Teachers can also receive personalized teaching suggestions provided by the teaching strategy adjustment suggestion model, such as optimization methods for the teaching content of a certain chapter, classroom interactive activity design ideas, etc., and can share teaching experiences and insights with other teachers in the interface and participate in teaching seminars. In addition, teachers can upload self-made teaching resources (such as teaching courseware, teaching videos, etc.) through the interface to enrich the platform's education data resource library. At the same time, they can also query the high-quality teaching resources already on the platform to provide more reference materials for teaching preparation.

[0039] Interactive interface for managers: provides a comprehensive analysis and decision-making support interface for medical management personnel on medical institution operation data. The interface displays the overall medical service quality statistical report of the hospital, including detailed data such as service quality scores of each department, summary of patient satisfaction survey results, and medical resource utilization efficiency indicators (such as bed turnover rate, medical equipment utilization rate, etc.). Managers can find potential problems and optimization space in hospital operation management through data screening and comparative analysis. In terms of resource allocation, managers can use the results of the medical resource allocation optimization model to make decision-making operations such as bed allocation adjustment, medical staff scheduling and medical equipment procurement planning, and view the effect prediction and actual feedback data after the decision is implemented in real time. For education managers, the interactive interface presents an overview of the overall situation of education in the school or region, such as comparison of teaching quality evaluation results of various subjects, distribution of students' overall learning status, and analysis of education resource input and output. Based on these data, managers can formulate education policies, plan education resource allocation plans (such as teacher training plans, teaching facility construction plans, etc.), and conduct macro-control and dynamic evaluation of the education development strategy of the school or region to ensure the sustainable development of education and effective guarantee of education equity.

[0040] General functions: The interactive interface of all user roles supports the system setting function. Users can set parameters such as data display format (such as data table style, chart color style, etc.), query conditions (such as custom data screening range, time interval selection, etc.), and warning thresholds (such as the lower limit of medical service quality indicators, student academic performance warning line, etc.) according to their own usage habits and needs. At the same time, the interactive interface has a data upload function. In addition to the above-mentioned teachers uploading teaching resources, medical personnel can also upload special case data (such as rare disease cases, difficult and complicated disease treatment process data, etc.). These uploaded data will be stored in the data storage module after the pre-processing process of the data acquisition module, further enriching the data resources of the platform system, improving the supervision and diagnosis capabilities of the platform system in special situations and complex scenarios, and promoting the continuous self-improvement and development of the platform system.

[0041] In a third aspect, an embodiment of the present application provides an electronic device, which includes a memory for storing one or more programs and a processor. When the one or more programs are executed by the processor, a system as described in any one of the first aspects above is implemented.

[0042] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a system as described in any one of the first aspects above.

[0043] The embodiments of the present invention have at least the following advantages or beneficial effects:

[0044] 1. The data integration and comprehensive analysis involved in the present invention, breaking the data silos in the medical field, successfully integrates the medical records, medical images and physiological monitoring data scattered in multiple sources such as hospital information management system (HIS), medical image storage and transmission system (PACS) and physiological monitoring equipment. For example, a patient's admission record, medical history record, CT image, electrocardiogram data, etc. are fully integrated, avoiding the tediousness of doctors switching queries between different systems, and providing panoramic information of patients for medical decision-making.

[0045] 2. The present invention involves the integration of teaching resource data, student learning behavior data and teaching evaluation data in the field of education. For example, the learning time and course resource clicks of students on the online learning platform are combined with the grade data and teacher evaluation data in the school's teaching management system, so that educators can understand the teaching process and student learning status from multiple dimensions, and are no longer limited to the one-sided cognition of a single data source.

[0046] 3. The multimodal data collaborative analysis involved in the present invention uses advanced multimodal data fusion technology to explore the potential correlation between different modal data. In medical treatment, the visual features of medical images complement the semantic information such as symptom descriptions and disease diagnosis in medical records. For example, in tumor diagnosis, the fusion analysis of tumor morphology, size and other features displayed in the image with text information such as the patient's family history and symptom development process in the medical record can improve the accuracy and comprehensiveness of disease diagnosis and reduce the risk of misdiagnosis and missed diagnosis.

[0047] 4. The present invention relates to the combination of student learning behavior data and teaching resource characteristics in education. For example, analyzing the relationship between students' preferences for a certain type of teaching resources (such as animation demonstration courseware) and learning behaviors (such as learning time and answer accuracy) can provide a basis for optimizing teaching resources and personalized learning recommendations, thereby improving teaching effectiveness.

[0048] 5. The present invention involves the field of precise supervision and diagnosis of medical care, and the precise evaluation of medical service quality: the evaluation model is built based on the fusion of multimodal data, and comprehensively considers the data of each link of the patient's medical process and the treatment effect data. It can accurately identify bottlenecks and problem points in medical services, such as finding that the average waiting time of patients in a certain department is too long, and the recurrence rate of a certain disease after treatment is high, thereby prompting medical institutions to improve service processes in a targeted manner, enhance medical technology levels, and improve patient satisfaction.

[0049] 6. Intelligent disease diagnosis assistance involved in the present invention: Utilizing a deep learning model to analyze fused feature vectors, complex disease feature patterns can be learned. For example, in the diagnosis of cardiovascular diseases, in addition to conventional electrocardiogram and cardiac ultrasound image analysis, it also combines the patient's lifestyle (inferred from data such as education and professional experience), family medical history and other factors to provide doctors with a more comprehensive diagnostic reference, assisting doctors in making more accurate diagnostic decisions, especially for the diagnosis of some atypical symptoms or complex diseases.

[0050] 7. The scientific optimization of medical resource allocation involved in the present invention: through the analysis of integer programming algorithm combined with multimodal data, it can be reasonably allocated according to the actual patient needs and the current status of medical resources. For example, in the high-incidence season of influenza, according to the patient flow and disease distribution, the number of beds in the respiratory department, the scheduling of medical staff and the preparation of related medical equipment are dynamically adjusted to improve the utilization efficiency of medical resources, avoid idle or excessive resources, and ensure the timeliness and quality of medical services.

[0051] 8. The present invention relates to the field of education and real-time monitoring of teaching quality: with the help of the model constructed by principal component analysis and linear regression, it is possible to track the changes in multi-dimensional data in the teaching process in real time. Timely discover the fluctuation factors of teaching quality, such as the changes in students' learning effects after the adjustment of teachers' teaching methods, the impact of teaching resource updates on students' learning interests, etc. Education managers can take measures quickly based on this, such as organizing teacher training, updating the teaching resource library, etc., to ensure the stable improvement of teaching quality.

[0052] 9. The present invention involves personalized analysis of student learning status: hierarchical clustering and association rule mining algorithms are used to accurately classify students' learning status categories and deeply explore the relationship between learning behavior and learning effect. Provide each student with a personalized learning status report to help students understand their learning strengths and weaknesses. For example, if a student is found to have difficulty learning the geometry part of mathematics, geometry learning special practice resources and tutoring courses can be recommended to achieve teaching in accordance with students' aptitude and promote students' autonomous learning and all-round development.

[0053] 10. The present invention involves precise adjustment of teaching strategies: Based on the model constructed by expert system and collaborative filtering algorithm, combined with the experience of experts in the field of education and the successful experience of similar teaching scenarios, it provides teachers with precise suggestions for adjusting teaching strategies. For example, when students in a class have difficulty understanding in English reading teaching, the model can recommend reading extension activities and layered reading teaching methods adopted by other similar classes, helping teachers optimize teaching strategies and improve classroom teaching efficiency and student academic performance.

[0054] 11. The user experience and decision support provided by the present invention are convenient for user interaction, and personalized interactive interfaces are designed for medical personnel, educators, and managers. Medical personnel can quickly and easily obtain comprehensive patient data and auxiliary diagnostic information during diagnosis and treatment, thereby improving work efficiency. For example, doctors can quickly query patients' medical history, imaging examination results, and fusion diagnostic recommendations during outpatient visits, thus reducing patient waiting time.

[0055] 12. The educators involved in the present invention can easily view the learning status of students in the class and the teaching quality evaluation results, and obtain timely suggestions for adjusting teaching strategies, which is convenient for teaching management and personalized tutoring. For example, teachers can quickly understand the students' mastery of the knowledge in this class after class, and adjust the teaching focus of the next class according to the platform's suggestions.

[0056] 13. The managers involved in the present invention can fully grasp the operation data of medical institutions or educational institutions through the interactive interface, and make macro decisions and resource allocation. For example, hospital managers can formulate procurement plans and personnel arrangements based on the results of medical resource allocation optimization, and education managers can formulate education development plans and resource allocation plans based on teaching quality data.

[0057] 14. Data-driven decision support involved in the present invention Whether in the medical or educational fields, the data visualization reports, analysis results and decision-making recommendations provided by the platform system are based on a large amount of integrated multimodal data and accurate model analysis. This makes decision-making more scientific and reasonable, and reduces the subjectivity and limitations of human experience judgment. For example, education policy makers can formulate education equity policies that are more in line with actual needs based on the overall regional education data provided by the platform, and promote the balanced distribution of education resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0059] Figure 1 It is a principle block diagram of the present invention;

[0060] Figure 2 A structural block diagram of an electronic device provided by an embodiment of the present invention.

[0061] Description of reference numerals: 101, memory; 102, processor; 103, communication interface. DETAILED DESCRIPTION

[0062] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0063] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0064] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0065] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the term "comprise" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0066] In the description of the embodiments of the present invention, "plurality" means at least 2.

[0067] In the description of the embodiments of the present invention, it is also necessary to explain that, unless otherwise clearly specified and limited, the terms "set", "install", "connect", and "connect" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal connection of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0068] Example:

[0069] 1. Data collection and preprocessing

[0070] Medical data collection and preprocessing

[0071] The medical data collection submodule of the data collection module is connected to the hospital's HIS system. Through SQL query statements, the basic information of 5,000 patients in the past year, such as name, age, gender, contact information, and detailed medical record text data, including admission records, daily medical records, surgical records, discharge summaries, etc., was accurately extracted. For medical record text data, redundant punctuation, repeated symptom descriptions, and some meaningless doctor's handwritten comments were removed in the preprocessing stage, and the text was uniformly converted to UTF-8 encoding format.

[0072] By connecting to the PACS system and following the DICOM standard protocol, we successfully acquired various medical imaging data of these 5,000 patients, totaling about 100,000 images, including X-rays, CT scans, MRI images, etc. During the acquisition process, we conducted a preliminary quality check on the image data and eliminated some blurred or damaged images caused by equipment failure or shooting errors. For the remaining images, we performed image enhancement processing, such as adjusting the contrast and brightness, to make the lesion area clearer and more discernible.

[0073] A real-time connection was established with the physiological monitoring equipment to receive the patient's electrocardiogram (ECG), blood pressure, blood oxygen saturation and other physiological monitoring data during hospitalization. The data collection frequency was set to once per minute, and a total of about 5 million physiological monitoring data points were collected. During preprocessing, a data smoothing algorithm was used to remove noise data caused by patient movement or short-term interference from the equipment, and the data collected by different devices were accurately synchronized according to the timestamp.

[0074] Educational data collection and preprocessing

[0075] The educational data collection submodule interacts with the online learning platform of educational institutions through the API interface. The learning behavior data of 3,000 students in the past semester were obtained, including the time when students logged into the platform, the name of the course, the learning time, the pause, fast forward, and rewind operation records in the course video, and the posting and reply content in the course discussion area. After data cleaning, abnormal click records and duplicate records caused by network failures were removed, and the data was organized into a structured table format.

[0076] The course schedule information of students, such as course name, lecturer, class time, classroom location, etc., as well as students' performance data, including regular homework performance, unit test performance, final exam performance, etc., and teacher evaluation data, such as teaching attitude, teaching method, teaching effect and other evaluation index scores, were extracted from the academic management system. Outlier detection was performed on the performance data, and a small number of performance data that were obviously deviated from the normal range due to input errors or special circumstances were corrected. -For teaching resource data, about 5,000 teaching courseware, teaching videos and other resources were directly extracted from the internal resource library of the educational institution. Metadata of teaching resources, such as resource title, introduction, upload time, subject, applicable grade and other information, were extracted. In the preprocessing stage, teaching resources were classified and sorted according to subjects and grades, and a resource directory was established to facilitate subsequent query and management.

[0077] (II) Data storage

[0078] Structured medical data, such as the patient basic information table and medical record information table, are stored in the MySQL relational database. In the patient basic information table, the patient ID is set as the primary key, including fields such as name, age, and gender; in the medical record information table, the medical record ID is the primary key, associated with the patient ID, and includes fields such as symptom description, diagnosis results, and treatment plans. By creating indexes, such as creating a general index on the name field of the patient basic information table and an index on the diagnosis result field of the medical record information table, the data query speed is improved.

[0079] Unstructured data such as medical imaging data, physiological monitoring data, and educational teaching videos and audio are stored in the HDFS distributed file system. For medical imaging data, a folder hierarchy is established according to patient ID and image type for storage. For example, all X-rays of patient 1 are stored in the " / medical_images / patient1 / X-ray" folder, and each image file is stored with its unique file name (such as patient1_X-ray_20240101.dcm). At the same time, external tables are established in the Hive data warehouse to manage the metadata of unstructured data for data query and analysis. For example, a medical_image_metadata table is created, which contains fields such as image file path, patient ID, image type, shooting time, etc. This table can be used to quickly locate and retrieve the required medical imaging data.

[0080] Structured data in the field of education, such as student score tables and course information tables, are also stored in MySQL relational databases. The student score table uses the student ID and course ID as the joint primary key, and contains fields such as grades and test times; the course information table uses the course ID as the primary key, and contains fields such as course names and instructors. Corresponding indexes are also established. For example, an index is established on the student ID field in the student score table to facilitate the query of all grade information for a specific student. (III) Multimodal data fusion

[0081] Feature extraction

[0082] For medical imaging data in the medical field, a convolutional neural network architecture based on residual network (ResNet) is used for feature extraction. Taking lung CT images as an example, the images are input into the ResNet model. After being processed by multiple convolutional layers and pooling layers, the model automatically learns the nodule features, texture features, and lung structure features in the lung images, and finally outputs a 512-dimensional feature vector. For medical record text data, the pre-trained BERT model is used for feature extraction. The BERT model converts each word in the medical record text into a 768-dimensional vector representation, and then uses a multi-layer Transformer structure to perform semantic modeling on the text, extracting key semantic features such as disease name, symptom description, and treatment process, and outputs a 1024-dimensional feature vector. The medical image feature vector and the medical record text feature vector are concatenated to obtain a 1536-dimensional multimodal feature representation of medical data. - In the field of education, for student learning behavior data, the K-Means clustering algorithm is used to divide students' learning behavior patterns into three categories: active, moderately stable, and passive and inefficient. For proactive students, their features are extracted as long study time, active participation in course discussions, high quality of homework completion, etc. For teaching resource data, the TF-IDF algorithm is used to extract the theme keyword features of the resources. For example, the keywords of a physics teaching courseware include "Newton's law", "friction", "kinetic energy theorem", etc. The student learning behavior feature vector is integrated with the teaching resource feature vector to form a multimodal feature set of educational data.

[0083] Fusion Algorithm

[0084] A multimodal fusion network based on the attention mechanism is adopted. The multimodal feature representation of medical data and the multimodal feature set of educational data are used as input. The network automatically focuses on the feature information that is more critical to the current supervision and diagnosis tasks by calculating the attention weights between different modal features. For example, when assisting in disease diagnosis and involving specific educational background factors (such as the patient's occupation and learning experience may affect the cause of the disease), the network will give a certain weight to the relevant educational features, and work together with the medical features to generate a fusion feature vector. In this embodiment, for a patient with cardiovascular disease who is engaged in high-intensity mental work (inferred from education and occupational experience data), the fusion network will appropriately increase the weight of his work pressure-related educational features when analyzing his disease data, and work together with cardiovascular imaging features, family history in medical records and other medical features to generate a more targeted fusion feature vector, providing a more comprehensive reference for disease diagnosis.

[0085] 3. Supervision and diagnosis

[0086] Medical supervision and diagnosis submodule

[0087] Medical service quality evaluation model: Based on the fusion feature vector, the support vector machine (SVM) algorithm is used to train the historical medical data of the above 5,000 patients. The model comprehensively considers various data in the entire medical process of patients from registration to discharge, such as waiting time, examination and testing process duration, treatment cost details, medical staff response timeliness, etc., as well as the final treatment effect indicators (such as cure rate, improvement rate, recurrence rate, etc.). The hyperplane constructed by SVM classifies medical cases with different service quality levels. For example, the service quality is divided into four levels: excellent, good, medium, and poor. After training and evaluation, it was found that the average waiting time for patients in a certain department was long, resulting in an overall service quality rating of medium. Based on this result, the hospital management department optimized the registration and treatment process of the department, shortened the patient waiting time, and in the subsequent evaluation, the service quality was significantly improved.

[0088] Disease diagnosis auxiliary model: A disease diagnosis auxiliary model is constructed using a deep belief network (DBN). The fused feature vector is input into the DBN, and the model mines the deep relationship between disease symptoms, imaging features, physiological monitoring data, and related educational background factors through unsupervised pre-training and supervised fine-tuning. For example, when a patient has unexplained chest pain symptoms, in addition to analyzing his electrocardiogram, cardiac ultrasound images and other medical data, the model also combines factors such as the patient's long-term high-intensity work (inferred from education and career experience data) and lack of exercise to improve the accuracy of diagnosis of cardiovascular diseases. Based on the diagnostic suggestions provided by the model, the doctor further arranged relevant examinations, and finally diagnosed coronary heart disease and developed a personalized treatment plan.

[0089] Medical resource allocation optimization model: The integer programming algorithm is used to construct a medical resource allocation optimization model. The model takes the patient flow data of each department of the hospital (including outpatient volume and inpatient volume), disease type distribution data, status data of different medical resources (such as beds, number of medical staff, number of various medical equipment and frequency of use) and related information in the fusion feature vector as input. For example, during the peak influenza season, based on the patient flow and disease distribution, the model calculates that the respiratory department needs to add 10 beds, deploy 5 medical staff, and add 3 respiratory monitoring devices. The hospital management deployed resources according to this plan, effectively responded to the peak of influenza, and improved the overall utilization efficiency and benefits of medical resources.

[0090] Education Supervision and Diagnosis Submodule

[0091] Teaching quality monitoring model: The teaching quality monitoring model is constructed by combining principal component analysis (PCA) with linear regression. PCA first reduces the dimensionality of the multi-dimensional data in the fusion feature vector (such as teacher teaching behavior characteristics, student learning effect characteristics, teaching resource utilization characteristics, etc.) to extract the main components, and then establishes the relationship model between teaching quality indicators (such as the improvement of students' average grades, students' teaching satisfaction survey scores, etc.) and the main components through linear regression. For example, in the mathematics teaching of a certain class, the model found that the teacher's single teaching method (analyzed from the teaching behavior characteristics) led to a decrease in students' learning interest, which in turn affected the improvement of students' average grades. Based on this result, the teaching management department organized teachers to participate in teaching method training. Teachers adopted a variety of teaching methods in subsequent teaching, such as group cooperative learning, mathematical experiments, etc., and students' learning interest and average grades were significantly improved.

[0092] Student learning status analysis model: A student learning status analysis model is constructed using hierarchical clustering and association rule mining algorithms. First, the hierarchical clustering algorithm divides the student group into different levels of learning status categories based on the student's learning behavior characteristics (such as learning time distribution, learning resource preferences, changes in answer accuracy, etc.). For example, students are divided into excellent and stable learning types, progressive learning types, and learning difficulties types. Then, the association rule mining algorithm mines the correlation between student learning behavior and learning effects within each category. Among students with learning difficulties, it is found that the association rules between "long homework completion time and high error rate" and "low class participation and difficulty in understanding specific knowledge points" often appear. Based on these analysis results, personalized learning resources are recommended for students with learning difficulties, such as tutoring videos for specific knowledge points, special exercise question banks, etc., and teachers are arranged for one-on-one tutoring to help students overcome learning difficulties and improve their academic performance.

[0093] Teaching strategy adjustment suggestion model: A teaching strategy adjustment suggestion model is constructed based on expert system and collaborative filtering algorithm. The expert system has built-in teaching experience and knowledge rules of experts in the field of education. According to the results of the teaching quality monitoring model and the student learning status analysis model, combined with the current teaching environment and educational goals, it generates preliminary teaching strategy adjustment suggestions. For example, in Chinese reading teaching, the expert system recommends strengthening reading extension training based on the slow improvement of students' reading ability. The collaborative filtering algorithm further considers the successful teaching experience of other similar classes and recommends the use of activities such as reading sharing sessions and reading check-ins. Teachers adjusted their teaching strategies based on these suggestions, and students' reading interest and reading ability were effectively improved. (V) User interaction

[0094] Medical staff interface

[0095] When doctors see patients in the outpatient clinic, they can quickly retrieve the patient's complete medical data on the patient data comprehensive query and diagnosis assistance interface by entering the patient ID. For example, when viewing the data of a patient with chest pain, the interface clearly displays the patient's medical record text details, including information such as hypertension and hyperlipidemia in the past medical history, medical images showing partial stenosis of the coronary arteries of the heart, physiological monitoring data showing large recent blood pressure fluctuations, and the results of the fused multimodal data feature analysis indicate a high risk of cardiovascular disease. During the diagnosis process, the doctor calls the prediction results provided by the disease diagnosis assistance model. The model predicts a high possibility of coronary heart disease based on the patient data and fused feature vectors, and provides similar case references and personalized diagnostic suggestions, such as further examination of myocardial enzyme spectrum. Based on this information, the doctor makes more accurate diagnostic decisions and records his own diagnostic ideas and treatment plans interactively with the platform. At the same time, doctors can also view the medical service quality evaluation report to understand the overall service level of their department and their own service quality ranking, so as to improve the service process and quality in a targeted manner.

[0096] Nurses use the patient care process management and nursing quality monitoring data display interface in their nursing work. For example, when caring for a postoperative patient, the interface displays the patient's nursing plan, including the time and method of postoperative wound care, infusion arrangements, etc. The nursing operation record query function allows nurses to view previous nursing operations, and the patient recovery progress tracking function displays the patient's vital signs recovery, wound healing, etc. in the form of charts. Nurses can also view medical resource allocation information related to nursing work, such as ward bed usage, nursing equipment deployment information, etc., to ensure the efficiency and safety of nursing work.

[0097] Educator Interface

[0098] After class, teachers log in to the panoramic analysis of students' learning and teaching strategy support interface to view individual learning status reports of students in the class they teach. For example, when checking a student's English learning situation, the report shows that the student performs well in vocabulary memory, but has deficiencies in listening comprehension and oral expression. The analysis of learning behavior characteristics shows that the student spends less time on listening practice resources and rarely participates in oral communication activities. The teaching quality evaluation report generated by the teaching quality monitoring model is presented in the form of intuitive charts. For example, the teaching effect trend chart of the English subject shows that the overall listening and oral scores of the class have not improved significantly. Teachers receive personalized teaching suggestions provided by the teaching strategy adjustment suggestion model, such as adding interactive sessions for listening training, organizing English oral corner activities, etc., and can share teaching experiences and insights with other teachers in the interface and participate in teaching seminars. In addition, teachers can select appropriate teaching resources from the internal resource library of educational institutions and upload them to the platform to enrich the platform's educational data resource library. At the same time, they can also query the high-quality English teaching resources already on the platform to provide more reference materials for teaching preparation.

[0099] Manager interface

[0100] Hospital managers can view the overall medical service quality statistics of the hospital through the comprehensive analysis and decision support interface of medical institution operation data. For example, the report shows that the patient satisfaction survey results of the whole hospital in a certain quarter were 85%. Among the service quality scores of various departments, the service quality of internal medicine was higher, while the waiting time of patients in surgery needed to be improved. Managers can gain an in-depth understanding of the operation of each department through data screening and comparative analysis. In terms of resource allocation, managers can use the results of the medical resource allocation optimization model to make decision-making operations such as bed allocation adjustment, medical staff scheduling arrangements, and medical equipment procurement planning. For example, according to the model recommendations, surgical beds were added to the surgical department, two experienced surgeons were recruited, and a new type of surgical equipment was purchased to improve the medical service capabilities and resource utilization efficiency of the surgical department.

[0101] Educational managers can view the comparison of teaching quality evaluation results of each subject, the distribution of students' overall learning status, and the analysis of educational resource input and output through the overall overview interface of the educational institution. For example, it is found that the teaching quality of mathematics in a certain grade is relatively low, and the proportion of students with learning difficulties is relatively high. Managers formulate educational policies based on these data, such as organizing special training for mathematics teachers, increasing investment in mathematics teaching resources, etc., and conduct macro-control and dynamic evaluation of the development strategies of educational institutions to ensure the sustainable development of education and effective protection of educational equity. It can be seen from this embodiment that the medical and educational information comprehensive supervision and diagnosis platform system with multimodal data fusion can effectively integrate data, accurately supervise and diagnose in actual applications, and provide convenient and valuable interactive experience for different users, which plays an important role in promoting the development of the medical and educational fields.

[0102] The embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by the processor 102, a system as in any one of the first aspects described above is implemented. If the function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a disk or an optical disk.

[0103] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

[0104] It will be apparent to those skilled in the art that the present application is not limited to the details of the exemplary embodiments described above, and that the present application can be implemented in other specific forms without departing from the spirit or essential features of the present application. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the present application is defined by the appended claims rather than the above description, and it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims be included in the present application. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

Claims

1. A comprehensive supervision and diagnosis platform system for medical and educational informationization based on multimodal data fusion, characterized by: It includes data acquisition module, data storage module, multimodal data fusion module, supervision and diagnosis module and user interaction module.

2. The multimodal data fusion integrated supervision and diagnosis platform system for medical and educational informationization according to claim 1 is characterized by: The data acquisition module includes a medical data acquisition submodule and an educational data acquisition submodule, wherein the medical data acquisition submodule can establish a connection with the hospital information management system, the medical image storage and transmission system and the physiological monitoring equipment to obtain medical record text data, medical image data and physiological monitoring data, and the submodule adopts a data verification algorithm to ensure the accuracy of the collected medical data; the educational data acquisition submodule is connected to the online learning platform and the school academic affairs management system, and is used to collect teaching resource data, student learning behavior data and teaching evaluation data, and remove duplicate, erroneous and incomplete data through data cleaning.

3. The multimodal data fusion integrated supervision and diagnosis platform system for medical and educational informationization according to claim 1 is characterized by: The data storage module adopts a distributed storage architecture. For structured data in the medical and educational fields, it uses a relational database to store and establishes a structured query index based on data attributes. For unstructured data such as medical images and teaching videos, it uses a non-relational database or distributed file system to store and establish a multi-dimensional index based on the data modality, source and timestamp. At the same time, data partitioning technology is used to partition the data according to different dimensions.

4. The multimodal data fusion integrated supervision and diagnosis platform system for medical and educational informationization according to claim 1 is characterized by: The feature extraction submodule in the multimodal data fusion module uses a convolutional neural network to extract texture, shape and other features from medical image data, uses natural language processing technology to extract semantic features from medical record text data, uses a data mining algorithm to extract behavioral pattern features from student learning behavior data, and uses text analysis technology to extract subject keywords and other features from teaching resource data.

5. The multimodal data fusion integrated supervision and diagnosis platform system for medical and educational informationization according to claim 1 is characterized by: The fusion algorithm submodule in the multimodal data fusion module adopts a multimodal fusion model based on deep learning. The model has the ability to automatically assign weights according to the importance of different modal features, so as to deeply fuse the feature vectors of different modalities in the medical and educational fields and generate a fused feature vector. The fusion process can dynamically adjust the weight of each modal feature according to the specific application scenario.

6. The multimodal data fusion integrated supervision and diagnosis platform system for medical and educational informationization according to claim 1 is characterized by: The supervision and diagnosis module includes a medical supervision and diagnosis submodule and an educational supervision and diagnosis submodule. The medical supervision and diagnosis submodule is based on a fusion feature vector and uses a machine learning algorithm to build a medical service quality assessment model, a deep learning algorithm to build a disease diagnosis auxiliary model, and an optimization algorithm to build a medical resource allocation optimization model.

7. The multimodal data fusion integrated supervision and diagnosis platform system for medical and educational informationization according to claim 1 is characterized by: The education supervision and diagnosis submodule in the supervision and diagnosis module constructs a teaching quality monitoring model based on the fusion feature vector and uses data mining and analysis technology, constructs a student learning status analysis model using a machine learning algorithm, and constructs a teaching strategy adjustment suggestion model using an expert system or an intelligent recommendation algorithm, and each model can be continuously optimized according to new data.

8. The multimodal data fusion integrated supervision and diagnosis platform system for medical and educational informationization according to claim 1 is characterized by: The user interaction module sets different operation interfaces for medical personnel, educators and managers. The medical personnel operation interface can realize multimodal display of patient data, viewing of diagnostic auxiliary information and data feedback upload; the educator operation interface can view the comprehensive learning situation of students in the class and receive teaching strategy suggestions; the manager operation interface can perform macro data statistical analysis and resource allocation decision-making, and each interface supports customized settings of system parameters.

9. The multimodal data fusion medical and educational information-based integrated supervision and diagnosis platform system according to claim 1 is characterized by: The user interaction module also has a data uploading function, allowing medical personnel to upload special case data and educators to upload self-made teaching resource data to enrich the platform data resource library, and the uploaded data must be stored in the data storage module after the preprocessing process of the data acquisition module.

10. The multimodal data fusion integrated supervision and diagnosis platform system for medical and educational informationization according to claim 1 is characterized by: The overall architecture of the platform system has the ability to manage data in a closed loop, that is, the entire process from data collection, storage, fusion, supervision and diagnosis to user interaction feedback data forms a complete data flow and processing cycle to continuously improve the performance and accuracy of the platform system.

Images