Deep learning-based teaching corpus construction method and system, and medium

Abstract

The invention discloses a teaching corpus construction method and system based on deep learning, and a medium, and belongs to the cross technical field of medical education, artificial intelligence and multi-modal data processing. The method comprises the following steps: a data acquisition and annotation stage: acquiring medical inquiry video data of different scenes; in the model training stage, a multi-modal deep learning model architecture is used for carrying out cross-modal alignment on multi-modal data; then semantic understanding and question classification are carried out based on the pre-training language model after fine tuning; in the corpus construction and optimization stage, multi-modal data are integrated, a corpus management system is built, dynamic updating and self-adaptive optimization are carried out on a corpus, meanwhile, a typical non-language posture library is constructed for a regional culture background, and teaching is carried out in combination with the corpus. According to the invention, in combination with medical professional knowledge, an efficient and intelligent medical inquiry video teaching corpus is constructed through deep fusion of natural language processing, computer vision, speech recognition and multi-modal data fusion technologies.

Description

Technical Field

[0001] The present invention relates to the interdisciplinary technical field of medical education, artificial intelligence, and multimodal data processing, and in particular to a method, system, and medium for constructing a teaching corpus based on deep learning. Background Art

[0002] Medical consultations constitute a critical link in a healthcare practitioner's initial contact with patients in clinical practice and are crucial for the accurate diagnosis of diseases. In the process of developing medical students' skills, training in medical consultation skills is fundamental to shaping their clinical thinking, improving their communication skills, and cultivating their humanistic care qualities. Currently, the cultivation of students' consultation skills relies primarily on traditional teaching methods such as simulation and bedside instruction. However, these methods have many limitations, such as limited patient resources, insufficient student practice opportunities, and a lack of effective feedback mechanisms. These issues lead to a disconnect between students' integration of theory and practice, insufficient communication skills, a lack of clinical thinking ability, and high psychological pressure, making it difficult for them to effectively adapt to the increasingly complex medical environment.

[0003] Deep learning constitutes a key branch of artificial intelligence. Its underlying architecture is artificial neural networks, which are capable of constructing and learning complex data patterns and are widely used in numerous fields. In medical education, deep learning technology, with its advantages such as automated knowledge extraction, personalized learning path planning, and automated assessment and feedback, is frequently used in the construction of virtual simulation scenarios, the generation of teaching resources, and the development of personalized learning assistants. Although existing AI-based teaching tools have brought innovation to traditional teaching models, they are limited in accuracy and reliability, lack the ability to understand complex concepts, and lack personalization and emotional interaction.

[0004] Building a deep learning-based medical consultation video corpus can alleviate problems such as the lack of patient resources in traditional education. Students can repeatedly practice medical consultation skills in a virtual environment, reducing their reliance on real patients and mitigating the risks associated with inexperience. This can also alleviate students' psychological stress when working with real patients, allowing them to focus on developing their medical consultation skills. The medical consultation video corpus covers a variety of scenarios, from simple consultations to complex communication, providing students with comprehensive practice material. The videos cover common symptoms and diagnostic scenarios for different diseases, and the data is sourced from authentic sources. This allows for a comprehensive development of students' medical consultation logic, communication skills, and humanistic care, ensuring consistent teaching quality and effectiveness, thereby promoting the intelligent development of medical education.

[0005] Current medical consultation education primarily relies on traditional classroom instruction or intelligent assistive devices, such as virtual reality (VR), augmented reality (AR), and mixed reality (MR). While these technologies have improved traditional medical consultation education to a certain extent, they still face numerous practical challenges, including high price and development costs, clunky operation, limited portability and comfort, and uneven resource allocation.

[0006] Although existing technologies have begun to attempt to incorporate artificial intelligence, their applications are mainly limited to the processing of single-modal data, and their processing capabilities for complex data patterns are still insufficient. Therefore, it has limitations in integrating multiple modal data such as voice, language, expression, and movement, and lacks the ability to deeply integrate and collaboratively analyze multimodal data. This leads to limitations in the utilization efficiency of teaching resources and teaching effectiveness. Current medical consultation teaching tools mainly rely on students to passively watch videos or participate in simulated teaching scenarios, which lack interactivity and have defects in personalized teaching. Existing technologies mostly use simple keyword matching or rule-based recommendation methods, and the accuracy and relevance of the recommendation results are low. Existing medical consultation teaching is mostly limited to traditional classroom teaching models, which are restricted by time and space and are difficult to meet the needs of telemedicine education. Summary of the Invention

[0007] The purpose of the present invention is to overcome the deficiencies of the prior art and provide a method, system and medium for constructing a teaching corpus based on deep learning.

[0008] The object of the present invention is achieved through the following technical solutions: In a first aspect, the present invention provides: a method for constructing a teaching corpus based on deep learning, comprising the following steps: In the data collection and annotation stage, medical consultation video data of different scenarios are collected from multiple data sources, and data preprocessing and data annotation are performed to obtain the first data; In the model training phase, audio stream data and image frame data are extracted from the first data. The audio stream data is input into a deep neural network model for speech recognition training to obtain text features. The image frame data is input into a convolutional neural network for image analysis to obtain image features. A multimodal deep learning model architecture is used to perform cross-modal alignment of speech features, text features, and image features. Semantic understanding and question classification are then performed based on a pre-trained language model fine-tuned for the medical field. During the corpus construction and optimization phase, we integrate multimodal data, build a corpus management system, and dynamically update and adaptively optimize the corpus. At the same time, we build a library of typical non-verbal gestures based on regional cultural backgrounds, and combine the corpus for teaching.

[0009] Preferably, the data collection and annotation stage further includes the following steps: Use a speech recognition model combined with acoustic model adaptation to perform dialect speech recognition and transcription, and map the transcribed text to standard medical terminology; At the same time, a posture recognition algorithm is used to extract the skeleton key points of the target to be identified in the video data, analyze the gestures, sitting posture and expression changes of the target to be identified, and associate the action features with the consultation dialogue at the corresponding timestamp to form an associated label; Screen the video clips to be annotated, remove video clips with blurred image quality and unclear audio, and classify them according to specific standards. Then, annotate each video clip with target information, use natural language processing methods to automatically extract the voice content in the video clips, and combine them with manual annotation for review and comparison, and finally complete the data annotation to obtain the first data.

[0010] Preferably, the model training stage further comprises the following steps: The audio stream data is subjected to noise reduction processing and then input into a deep neural network model for speech recognition training. The recognized speech content is converted into text, and the medical consultation dialogue text is generated to obtain text features; The image frame data is input into the convolutional neural network for image analysis, and the non-verbal communication information in the image frame is collected to obtain image features; Use a multimodal deep learning model architecture to align speech features, text features, and image features on the time axis to achieve cross-modal alignment; Through the pre-trained language model, the medical consultation dialogue is deeply semantically analyzed to extract key information in the conversation; based on the pre-trained speech model, a classification model is built, and classification labels are defined. The medical consultation dialogue is input into the classification model to automatically identify and classify it into the corresponding links.

[0011] Preferably, the corpus construction and optimization stage further includes the following steps: Convert data from different modalities into a compatible structured data format. Each video clip corresponds to a standard recording unit, which contains basic video information, analyzed diagnosis and treatment information, and interaction mode. Depending on the application scenario, a relational database or a non-relational database is used to build a corpus management system. The corpus management system introduces a full-text search mechanism to locate relevant video content by keywords, and uses machine learning algorithms to make personalized recommendations based on the user's historical search records and teaching objectives; Set the training cycle based on the amount of data and system performance; within each training cycle, retrain the deep learning model using new video data and existing corpus, and evaluate the trained model through cross-validation, accuracy, and recall; integrate the new knowledge generated by the trained model into the corpus, update the content of the corpus, and delete outdated or low-value knowledge according to the set elimination mechanism; dynamically adjust the model parameters and training strategies when teaching needs change.

[0012] Preferably, the multimodal deep learning model architecture is a multimodal Transformer architecture, the pre-trained language model is a BERT model, the speech recognition model is a Whisper model, and the posture recognition algorithm is a BlazePose algorithm; the skeletal key points include the head, shoulders, and elbows; the specific standards include disease type and diagnosis and treatment stage; the target information includes medical history collection, symptoms and signs, and diagnosis and treatment recommendations.

[0013] Preferably, it also includes an intelligent recommendation and personalized teaching stage, including the following steps: The system provides a learning platform to collect users' learning data in real time, analyze and process the learning data through data mining, and generate learning progress reports; by comparing the preset learning goals with the actual learning results, it can identify the weak links in the user's learning process; The system introduces a deep reinforcement learning algorithm to dynamically adjust recommendation strategies based on user learning behavior and feedback. The system tracks students' learning progress and knowledge gaps in real time and generates multi-dimensional learning assessment reports. Based on these reports, it dynamically recommends case resources. The system continuously optimizes recommended content based on user feedback and learning outcomes to form personalized teaching plans. According to the teaching content, interactive teaching scenes are generated for teaching.

[0014] Preferably, the teaching corpus construction method is used in any scenario of telemedicine education, medical skills assessment and evaluation, clinical training and continuing education.

[0015] A second aspect of the present invention provides: a teaching corpus construction system based on deep learning, characterized in that: used in any of the above-mentioned teaching corpus construction methods based on deep learning, comprising: A data collection and annotation module is used to collect medical consultation video data of different scenarios from multiple data sources, and perform data preprocessing and data annotation to obtain first data; A model training module is configured to extract audio stream data and image frame data from the first data, input the audio stream data into a deep neural network model for speech recognition training to obtain text features; input the image frame data into a convolutional neural network for image analysis to obtain image features; perform cross-modal alignment of speech features, text features, and image features using a multimodal deep learning model architecture; and then perform semantic understanding and question classification based on a pre-trained language model fine-tuned for the medical field; The corpus construction and optimization module is used to integrate multimodal data, build a corpus management system, and dynamically update and adaptively optimize the corpus. At the same time, it builds a typical non-verbal gesture library based on regional cultural backgrounds and combines the corpus for teaching.

[0016] The third aspect of the present invention provides: a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are loaded and executed by a processor, any of the above-mentioned teaching corpus construction methods based on deep learning is implemented.

[0017] The beneficial effects of the present invention are: 1) Combining medical expertise, through the deep integration of natural language processing (NLP), computer vision (CV), automatic speech recognition (ASR) and multimodal data fusion technology, an efficient and intelligent medical consultation video teaching corpus has been constructed, providing rich teaching resources for medical education.

[0018] 2) By using a pre-trained model based on the Transformer architecture, the present invention can realize speech recognition in multiple dialects and improve the accuracy of dialect recognition through acoustic model adaptation technology.

[0019] 3) This paper introduces a deep reinforcement learning (DRL) framework to monitor students’ learning progress, knowledge gaps, and interactive performance in real time, and dynamically recommends appropriate case resources based on this to better meet the learning needs of different students.

[0020] 4) Accurate matching of teaching resources: By tracking students' personalized learning progress, relevant consultation videos and learning resources are automatically recommended to meet students' diverse needs.

[0021] 5) Data-driven continuous optimization: As new videos continue to increase, the corpus has the ability to self-learn and optimize, keeping the content up to date.

[0022] 6) Teaching time and economic savings: The corpus supports distance education, allowing students to train at any time and place, saving teaching time and economic costs.

[0023] 7) Effectively improve medical consultation capabilities: Through natural interaction and increased practice opportunities, medical students' consultation thinking ability, communication skills and humanistic care literacy are effectively improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 Flowchart of the methodology for building a deep learning-based teaching corpus. DETAILED DESCRIPTION

[0025] The following will clearly and completely describe the technical solutions of the present invention in conjunction with the embodiments. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work shall fall within the scope of protection of the present invention.

[0026] The core technologies of this invention include a pre-trained model based on the Transformer architecture for multi-dialect speech recognition, a posture estimation algorithm for capturing patient body language and expressions, semantic understanding and classification combined with models such as BERT and Whisper, and personalized dynamic recommendations through a reinforcement learning framework. This corpus not only supports standardized medical consultation teaching but also adapts to dialects and cultural backgrounds in different regions, significantly improving medical students' clinical communication skills and awareness of humanistic care. It can be widely used in medical education, remote training, and clinical skills assessment, and has important teaching application value and research significance.

[0027] See Figure 1 The first aspect of the present invention provides: a method for constructing a teaching corpus based on deep learning, comprising the following steps: In the data collection and annotation stage, medical consultation video data of different scenarios are collected from multiple data sources, and data preprocessing and data annotation are performed to obtain the first data; In the model training phase, audio stream data and image frame data are extracted from the first data. The audio stream data is input into a deep neural network model for speech recognition training to obtain text features. The image frame data is input into a convolutional neural network for image analysis to obtain image features. A multimodal deep learning model architecture is used to perform cross-modal alignment of speech features, text features, and image features. Semantic understanding and question classification are then performed based on a pre-trained language model fine-tuned for the medical field. During the corpus construction and optimization phase, we integrate multimodal data, build a corpus management system, and dynamically update and adaptively optimize the corpus. At the same time, we build a library of typical non-verbal gestures based on regional cultural backgrounds, and combine the corpus for teaching.

[0028] In this embodiment, medical consultation video data is collected from different channels (such as hospitals, communities, clinics, simulated patient teaching, etc.) to ensure that the video content covers common symptoms and diagnostic scenarios of different diseases. The steps of data collection include: (1) Video collection: including real-life simulated medical consultations and the diagnosis process of real patients, ensuring that the collected scenes are representative, covering typical diseases in different departments (such as internal medicine, surgery, pediatrics, etc.), and especially including scenes containing dialect dialogues (such as Sichuan and Chongqing dialects, etc.).

[0029] (2) Data processing: A multilingual extended version of the Whisper model is used for dialect speech recognition and transcription. Acoustic model adaptation technology is combined to improve dialect recognition accuracy. Transcribed text is mapped to standard medical terminology to ensure the universality of teaching resources. MediaPipe's BlazePose algorithm is also used to extract key skeletal points of patients in the video. The patient's gestures, sitting posture, and facial expressions (such as frowning in pain and fidgeting in anxiety) are analyzed. The action features are then associated with the corresponding timestamps of the medical conversation to form "action-semantic" labels.

[0030] (3) Data annotation: Detailed annotation is performed for each video clip, including the following content: medical history collection, symptoms and signs, diagnosis and treatment recommendations, communication and interaction, etc. Annotation is performed through a combination of manual and semi-automatic methods. Using deep learning natural language processing (NLP) technology, the voice content in the video is automatically extracted and converted into text, reducing the workload of manual annotation and improving annotation accuracy and efficiency.

[0031] After the labeling is completed, deep learning technology is used to process and analyze the collected video data: (1) Speech recognition and processing: Use the deep neural network (DNN) model to recognize and convert the speech content in the video to ensure that the conversation content between the interviewer and the patient can be accurately extracted.

[0032] (2) Image and video analysis: The image content in the video is analyzed through a convolutional neural network (CNN) to identify the patient's facial expressions, postures, movements and other non-verbal communication information. The multimodal Transformer architecture is used to achieve cross-modal alignment of speech, text and image features to ensure data consistency.

[0033] (3) Semantic understanding and question classification: NLP technology performs in-depth semantic analysis on the language used in medical consultations, identifying key information such as the type of questions asked by doctors, the content of the diagnosis, and the patient's answers. The BERT-based medical field fine-tuning model performs in-depth semantic analysis on medical consultation dialogues, automatically classifying key links such as medical history collection and diagnostic reasoning.

[0034] After the deep learning analysis is completed, all the annotation and analysis results will be summarized and stored in the corpus. The construction of the corpus includes the following steps: (1) Multimodal data integration: Video data, text data (speech-to-text), image data (expressions, actions), and other different forms of data are unified and integrated and stored as structured data. Each video clip corresponds to a standardized recording unit, including: basic video information, analyzed diagnosis and treatment information, and the interaction mode between doctors and patients (such as questioning methods, patient reactions, etc.).

[0035] (2) Intelligent Corpus Management System: Using database technology to build an efficient corpus management system, it supports rapid retrieval, updating, and expansion of data. Through functions such as keyword search and intelligent recommendation, users can quickly find video content related to teaching objectives, thereby improving the efficiency of teaching resource utilization.

[0036] (3) Dynamic update and adaptive optimization: The corpus has the ability to self-learn and optimize. As new video data is continuously added, the deep learning model will be trained and updated regularly to keep the content of the corpus always up to date and can automatically adjust according to changes in teaching needs.

[0037] (4) Construction of intelligent management system: An intelligent management system is built using Elasticsearch technology, which supports multi-dimensional retrieval based on "dialect type + disease keywords + action tags". For example, teachers can quickly retrieve "abdominal pain cases of patients in Sichuan and Chongqing" and associate their body language data.

[0038] (5) Humanistic design: The corpus specifically annotates the patients’ regional cultural background (e.g., regional communication habits, differences between urban and rural patients), and constructs a “typical non-verbal gesture library.” For example, rural patients may be more inclined to use gestures to describe their symptoms, while urban patients rely more on verbal expression. By integrating dialect recognition with semantic associations of actions, the technical capabilities of medical students are improved, and their empathy and cross-cultural adaptability are strengthened, laying a solid foundation for clinical practice.

[0039] In some embodiments, the data collection and annotation stage further includes the following steps: Use a speech recognition model combined with acoustic model adaptation to perform dialect speech recognition and transcription, and map the transcribed text to standard medical terminology; At the same time, a posture recognition algorithm is used to extract the skeleton key points of the target to be identified in the video data, analyze the gestures, sitting posture and expression changes of the target to be identified, and associate the action features with the consultation dialogue at the corresponding timestamp to form an associated label; Screen the video segments to be annotated, remove the video segments with blurred picture quality and unclear audio, classify them according to specific criteria, then annotate the target information for each video segment, use natural language processing methods to automatically extract the speech content in the video segment, and combine it with manual annotation for review and comparison. Finally, complete the data annotation to obtain the first data.

[0040] In this embodiment, professional personnel and real patients are arranged to conduct medical consultations for video collection, covering typical diseases in multiple departments such as internal medicine, surgery, pediatrics, and obstetrics and gynecology; use high-definition cameras to record the medical consultation process of real patients in actual medical environments such as hospitals and community clinics, covering patients of different ages, genders, and conditions; specifically include scenarios with dialect conversations, such as the Sichuan-Chongqing dialect pronunciation of "stomachache" as "abdomen pain"; store the recorded videos, and the naming includes fields such as department, disease, and dialect type.

[0041] Data processing includes: (1) Speech transcription: Use the Whisper model to load pre-trained Chinese weights and fine-tune the acoustic model for different dialects; map the output text to standard medical terms, such as mapping the Sichuan-Chongqing dialect "having malaria" to "malaria".

[0042] (2) Skeletal key point extraction: Use the BlazePose algorithm of MediaPipe to extract the skeletal key points of the patient from the video in real time, including parts such as the head, shoulders, and elbows; based on the extracted skeletal key points, identify the patient's gesture actions (raising hands, pointing, etc.), evaluate the sitting posture state (leaning forward, leaning backward, etc.), and identify the expression changes (frowning when in pain, fidgeting when anxious, etc.).

[0043] (3) Association of action features and consultation conversations: Keep the extracted action features consistent with the timestamps of the consultation conversations; generate corresponding "action-semantic" labels according to the action features and the content of the consultation conversations, such as "fidgeting - anxious".

[0044] Data annotation includes: (1) collecting and organizing the video clips to be annotated, conducting preliminary screening, removing clips with blurry images and unclear audio, and classifying the video clips according to specific standards (such as disease type, diagnosis and treatment stage, etc.); (2) organizing a professional annotation team with a medical background to perform preliminary manual annotation on each video clip. The annotation content includes but is not limited to medical history collection (patient's basic information, medical history, etc.), symptoms and signs (pain location, cough degree, etc.), and diagnosis and treatment recommendations (diagnosis results, treatment plan, etc.); (3) using deep learning natural language processing (NLP) technology to develop a video speech recognition module, automatically convert the speech content in the video into text, generate a preliminary text transcript and perform preliminary proofreading on it to correct obvious recognition errors; (4) compare the automatically generated text transcript with the manual preliminary annotation results to check for omissions and ensure that the final annotation results are consistent with the video content.

[0045] In some embodiments, the model training phase further includes the following steps: The audio stream data is subjected to noise reduction processing and then input into a deep neural network model for speech recognition training. The recognized speech content is converted into text, and the medical consultation dialogue text is generated to obtain text features; The image frame data is input into the convolutional neural network for image analysis, and the non-verbal communication information in the image frame is collected to obtain image features; Use a multimodal deep learning model architecture to align speech features, text features, and image features on the time axis to achieve cross-modal alignment; Through the pre-trained language model, the medical consultation dialogue is deeply semantically analyzed to extract key information in the conversation; based on the pre-trained speech model, a classification model is built, and classification labels are defined. The medical consultation dialogue is input into the classification model to automatically identify and classify it into the corresponding links.

[0046] In this embodiment, non-verbal communication information includes facial expressions (happy, sad, etc.), actions (nodding, raising hands, etc.) in the image frame. Semantic analysis and question classification include the following steps: (1) Selecting the BERT model as the basis, fine-tuning it on the medical field corpus to make it better adapted to the medical context; (2) Using the fine-tuned BERT model to perform deep semantic analysis on the medical consultation dialogue to extract key information in the dialogue; such as the type of doctor's questions (open, closed questions, etc.), diagnosis content (disease name, symptom description, etc.), patient's answer (medical history information, etc.); (3) Building a classification model based on the BERT fine-tuned model to identify different links in the medical consultation dialogue; (4) Define classification labels, such as "medical history collection", "treatment plan discussion", etc.; input the medical consultation dialogue into the classification model, and automatically identify and classify it into the corresponding link. Example Doctor: Have you felt dizzy recently? Patient: Yes, I often feel dizzy in recent days, especially when I get up in the morning. First, the conversation text is annotated. For example, "dizziness" is marked as a symptom. The processed text is input into the fine-tuned BERT model. The model recognizes "dizziness" as a symptom and the doctor's question as an open-ended question. The model classifies this conversation as "medical history collection."

[0047] In some embodiments, the corpus construction and optimization stage further includes the following steps: Convert data from different modalities into a compatible structured data format. Each video clip corresponds to a standard recording unit, which contains basic video information, analyzed diagnosis and treatment information, and interaction mode. Depending on the application scenario, a relational database or a non-relational database is used to build a corpus management system. The corpus management system introduces a full-text search mechanism to locate relevant video content by keywords, and uses machine learning algorithms to make personalized recommendations based on the user's historical search records and teaching objectives; Set the training cycle based on the amount of data and system performance; within each training cycle, retrain the deep learning model using new video data and existing corpus, and evaluate the trained model through cross-validation, accuracy, and recall; integrate the new knowledge generated by the trained model into the corpus, update the content of the corpus, and delete outdated or low-value knowledge according to the set elimination mechanism; dynamically adjust the model parameters and training strategies when teaching needs change.

[0048] In this embodiment, data from different sources and formats (including videos, text, and images) are converted into a compatible structured data format to ensure the consistency of video, text, and image data on the timeline. A standardized recording unit is designed for each video clip, which contains the following information: basic video information: including video number, recording time, duration, etc.; diagnosis and treatment information obtained from analysis: covering diagnosis results, treatment plans, medication recommendations, etc.; interaction mode: involving the way doctors ask questions, patient reactions, etc.

[0049] When selecting a database, one should carefully decide whether to use a relational database or a non-relational database based on actual needs in order to build the most appropriate data storage solution and then establish a corpus management system. A reasonable database table structure should be designed to store video files and their metadata information to ensure orderly organization and efficient access of data. A full-text search function should be provided so that users can quickly locate relevant video content through keywords. At the same time, efficient search engine technology (such as Elasticsearch) should be used to improve retrieval efficiency. Based on the user's historical search records and teaching objectives, machine learning algorithms (such as collaborative filtering, content recommendation, etc.) should be used to achieve personalized recommendations to enhance the user experience.

[0050] Based on the amount of data and system performance, set a reasonable training cycle (such as weekly, monthly, etc.); within each training cycle, use new video data and existing corpus to retrain the deep learning model, and evaluate the trained model through cross-validation, accuracy, recall and other indicators; integrate the new knowledge generated by the trained model into the corpus and update the content of the corpus; eliminate outdated or low-value knowledge according to the set elimination mechanism (such as timestamp, frequency of use, etc.) to maintain the efficiency of the corpus; monitor changes in teaching needs in real time, and dynamically adjust the model parameters and training strategies based on the monitored changes in needs to make the model better adapt to new teaching needs.

[0051] Elasticsearch was selected as the core retrieval technology, supplemented by other related technologies such as SpringBoot to build an intelligent management system; an Elasticsearch index was designed, including fields such as dialect type, disease keywords, and action tags, and the processed data was imported into the index; a multi-dimensional retrieval interface was designed to support users entering dialect types, disease keywords, and action tags for combined queries.

[0052] Humanistic design: Establish a "typical non-verbal gesture library" to describe in detail the meaning of various gestures and their frequency of use in different cultural contexts; combine the non-verbal gesture library to analyze the relationship between patient actions and semantics, and establish a movement semantic mapping model; integrate dialect recognition results with non-verbal gesture data to form a comprehensive patient expression analysis system.

[0053] In some embodiments, the multimodal deep learning model architecture is a multimodal Transformer architecture, the pre-trained language model is a BERT model, the speech recognition model is a Whisper model, and the posture recognition algorithm is a BlazePose algorithm; the skeletal key points include the head, shoulders, and elbows; the specific criteria include disease type and treatment stage; the target information includes medical history collection, symptoms and signs, and treatment recommendations.

[0054] In some embodiments, the process further includes an intelligent recommendation and personalized teaching phase, including the following steps: The system provides a learning platform to collect users' learning data in real time, analyze and process the learning data through data mining, and generate learning progress reports; by comparing the preset learning goals with the actual learning results, it can identify the weak links in the user's learning process; The system introduces a deep reinforcement learning algorithm to dynamically adjust recommendation strategies based on user learning behavior and feedback. The system tracks students' learning progress and knowledge gaps in real time and generates multi-dimensional learning assessment reports. Based on these reports, it dynamically recommends case resources. The system continuously optimizes recommended content based on user feedback and learning outcomes to form personalized teaching plans. According to the teaching content, interactive teaching scenes are generated for teaching.

[0055] In this embodiment, the system collects students' learning data in real time through the learning platform, including but not limited to learning time, homework completion status, test scores, etc.; uses data mining technology to process and analyze the collected data to generate students' learning progress reports; and identifies students' weak links in the learning process by comparing preset learning goals with actual learning results.

[0056] The system integrates deep learning-based speech recognition and natural language processing technologies to create highly realistic interactive teaching scenarios. Based on the teaching content, interactive teaching scenarios are generated. For example, in a simulated medical consultation scenario, students communicate with a virtual patient through voice.

[0057] In some embodiments, the teaching corpus construction method is used in any scenario of telemedicine education, medical skills assessment and evaluation, clinical training and continuing education.

[0058] In this embodiment, the system will be improved in real time. The system combines with the Unity3D engine to develop a virtual patient interaction module, and students can have dialect conversations with virtual patients through voice; the system automatically scores students based on their questioning logic, communication skills, etc.; the system provides real-time improvement suggestions and optimization plans based on the scoring results; students conduct repeated training based on feedback until they reach the ideal level; through real-time feedback and improvement, students' medical questioning skills and clinical decision-making capabilities are continuously improved.

[0059] In the telemedicine education scenario, teachers upload medical consultation video teaching materials through the system and set up course schedules; students log in to the system through the Internet and select corresponding courses to study; after watching the video, students use the simulated consultation function provided by the system for practical training; teachers can monitor students' learning progress in real time and provide online Q&A and feedback.

[0060] In the medical skills assessment and evaluation scenario, the system sets up different simulated consultation scenarios according to the assessment requirements, and students enter the simulated scenarios to perform consultation operations; the system records students' diagnostic thinking, communication skills, clinical decision-making and other performances, and generates an evaluation report; teachers provide targeted guidance and improvement suggestions to students based on the evaluation report.

[0061] In clinical training and continuing education scenarios, corresponding consultation videos are recommended based on the learning needs of on-the-job medical staff. Medical staff can watch videos through the system to learn the latest diagnosis and treatment methods and techniques; apply the knowledge they have learned to actual clinical work to improve their clinical skills; the system regularly updates the consultation video library to ensure the cutting-edge content.

[0062] A second aspect of the present invention provides: a teaching corpus construction system based on deep learning, characterized in that: used in any of the above-mentioned teaching corpus construction methods based on deep learning, comprising: A data collection and annotation module is used to collect medical consultation video data of different scenarios from multiple data sources, and perform data preprocessing and data annotation to obtain first data; A model training module is configured to extract audio stream data and image frame data from the first data, input the audio stream data into a deep neural network model for speech recognition training to obtain text features; input the image frame data into a convolutional neural network for image analysis to obtain image features; perform cross-modal alignment of speech features, text features, and image features using a multimodal deep learning model architecture; and then perform semantic understanding and question classification based on a pre-trained language model fine-tuned for the medical field; The corpus construction and optimization module is used to integrate multimodal data, build a corpus management system, and dynamically update and adaptively optimize the corpus. At the same time, it builds a typical non-verbal gesture library based on regional cultural backgrounds and combines the corpus for teaching.

[0063] The third aspect of the present invention provides: a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are loaded and executed by a processor, any of the above-mentioned teaching corpus construction methods based on deep learning is implemented.

[0064] This invention uses deep learning technology to comprehensively collect, analyze and process data from multiple modalities to ensure the accuracy of recognition and the versatility of teaching resources, providing more comprehensive and accurate teaching resources for medical consultation teaching.

[0065] This method uses deep neural network technology to accurately extract the content of conversations between the interviewer and the patient, and uses convolutional neural networks to identify non-verbal communication information such as the patient's facial expressions, posture, and movements. By deeply training the model and specifically annotating the patient's regional cultural background and supporting dialect systems, a more interactive and realistic teaching environment is created.

[0066] This invention uses Elasticsearch technology to build an intelligent management system, supporting multi-dimensional retrieval based on "dialect type + disease keywords + action tags". At the same time, it combines the deep reinforcement learning framework to track students' individual learning differences in real time, achieving the efficiency and accuracy of the intelligent recommendation system, enabling students to quickly obtain the learning resources that best suit them, thereby improving learning efficiency and teaching effectiveness.

[0067] The medical consultation video teaching corpus of the present invention is not only suitable for traditional medical classroom teaching, but can also be widely used in the fields of telemedicine education, medical skills assessment and evaluation, clinical training and continuing education. Teachers can conduct remote teaching and students can conduct simulation training at any time and any place, which improves the effect of distance education, provides students with more learning opportunities, and broadens the application scenarios of medical consultation teaching.

[0068] The foregoing description is merely a preferred embodiment of the present invention. It should be understood that the present invention is not limited to the form disclosed herein and should not be construed as excluding other embodiments. Rather, the present invention can be used in various other combinations, modifications, and environments and can be modified within the scope of the concept described herein through the above teachings or techniques or knowledge in the relevant field. Modifications and variations made by those skilled in the art that do not depart from the spirit and scope of the present invention are intended to be protected by the appended claims.

Claims

1. A method for constructing a teaching corpus based on deep learning, characterized by: The following steps are involved: In the data collection and annotation stage, medical consultation video data of different scenarios are collected from multiple data sources, and data preprocessing and data annotation are performed to obtain the first data; In the model training phase, audio stream data and image frame data are extracted from the first data. The audio stream data is input into a deep neural network model for speech recognition training to obtain text features. The image frame data is input into a convolutional neural network for image analysis to obtain image features. A multimodal deep learning model architecture is used to perform cross-modal alignment of speech features, text features, and image features. Semantic understanding and question classification are then performed based on a pre-trained language model fine-tuned for the medical field. During the corpus construction and optimization phase, we integrate multimodal data, build a corpus management system, and dynamically update and adaptively optimize the corpus. At the same time, we build a library of typical non-verbal gestures based on regional cultural backgrounds, and combine the corpus for teaching.

2. The method for constructing a teaching corpus based on deep learning according to claim 1, characterized in that: The data collection and annotation stage further includes the following steps: Use a speech recognition model combined with acoustic model adaptation to perform dialect speech recognition and transcription, and map the transcribed text to standard medical terminology; At the same time, a posture recognition algorithm is used to extract the skeleton key points of the target to be identified in the video data, analyze the gestures, sitting posture and expression changes of the target to be identified, and associate the action features with the consultation dialogue at the corresponding timestamp to form an associated label; Screen the video clips to be annotated, remove video clips with blurred image quality and unclear audio, and classify them according to specific standards. Then, annotate each video clip with target information, use natural language processing methods to automatically extract the voice content in the video clips, and combine them with manual annotation for review and comparison, and finally complete the data annotation to obtain the first data.

3. The method for constructing a teaching corpus based on deep learning according to claim 1, characterized in that: The model training phase also includes the following steps: The audio stream data is subjected to noise reduction processing and then input into a deep neural network model for speech recognition training. The recognized speech content is converted into text, and the medical consultation dialogue text is generated to obtain text features; The image frame data is input into the convolutional neural network for image analysis, and the non-verbal communication information in the image frame is collected to obtain image features; Use a multimodal deep learning model architecture to align speech features, text features, and image features on the time axis to achieve cross-modal alignment; Through the pre-trained language model, the medical consultation dialogue is deeply semantically analyzed to extract key information in the conversation; based on the pre-trained speech model, a classification model is built, and classification labels are defined. The medical consultation dialogue is input into the classification model to automatically identify and classify it into the corresponding links.

4. The method for constructing a teaching corpus based on deep learning according to claim 1, characterized in that: The corpus construction and optimization phase also includes the following steps: Convert data from different modalities into a compatible structured data format. Each video clip corresponds to a standard recording unit, which contains basic video information, analyzed diagnosis and treatment information, and interaction mode. Depending on the application scenario, a relational database or a non-relational database is used to build a corpus management system. The corpus management system introduces a full-text search mechanism to locate relevant video content by keywords, and uses machine learning algorithms to make personalized recommendations based on the user's historical search records and teaching objectives; Set the training cycle based on the amount of data and system performance; within each training cycle, retrain the deep learning model using new video data and existing corpus, and evaluate the trained model through cross-validation, accuracy, and recall; integrate the new knowledge generated by the trained model into the corpus, update the content of the corpus, and delete outdated or low-value knowledge according to the set elimination mechanism; dynamically adjust the model parameters and training strategies when teaching needs change.

5. The method for constructing a teaching corpus based on deep learning according to claim 2, characterized in that: The multimodal deep learning model architecture is a multimodal Transformer architecture, the pre-trained language model is a BERT model, the speech recognition model is a Whisper model, and the posture recognition algorithm is a BlazePose algorithm; the skeletal key points include the head, shoulders, and elbows; the specific standards include disease type and diagnosis and treatment stage; the target information includes medical history collection, symptoms and signs, and diagnosis and treatment recommendations.

6. The method for constructing a teaching corpus based on deep learning according to any one of claims 1 to 5, characterized in that: It also includes the intelligent recommendation and personalized teaching stage, including the following steps: The system provides a learning platform to collect users' learning data in real time, analyze and process the learning data through data mining, and generate learning progress reports; by comparing the preset learning goals with the actual learning results, it can identify the weak links in the user's learning process; The system introduces a deep reinforcement learning algorithm to dynamically adjust recommendation strategies based on user learning behavior and feedback. The system tracks students' learning progress and knowledge gaps in real time and generates multi-dimensional learning assessment reports. Based on these reports, it dynamically recommends case resources. The system continuously optimizes recommended content based on user feedback and learning outcomes to form personalized teaching plans. According to the teaching content, interactive teaching scenes are generated for teaching.

7. The method for constructing a teaching corpus based on deep learning according to any one of claims 1 to 5, characterized in that: The teaching corpus construction method can be used in any scenario of telemedicine education, medical skills assessment and evaluation, clinical training and continuing education.

8. A teaching corpus construction system based on deep learning, characterized by: The method for constructing a teaching corpus based on deep learning according to any one of claims 1 to 7 comprises: A data collection and annotation module is used to collect medical consultation video data of different scenarios from multiple data sources, and perform data preprocessing and data annotation to obtain first data; A model training module is configured to extract audio stream data and image frame data from the first data, input the audio stream data into a deep neural network model for speech recognition training to obtain text features; input the image frame data into a convolutional neural network for image analysis to obtain image features; perform cross-modal alignment of speech features, text features, and image features using a multimodal deep learning model architecture; and then perform semantic understanding and question classification based on a pre-trained language model fine-tuned for the medical field; The corpus construction and optimization module is used to integrate multimodal data, build a corpus management system, and dynamically update and adaptively optimize the corpus. At the same time, it builds a typical non-verbal gesture library based on regional cultural backgrounds and combines the corpus for teaching.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions. When the computer-executable instructions are loaded and executed by the processor, the method for constructing a teaching corpus based on deep learning as described in any one of claims 1 to 7 is implemented.

Images