AI-based method and system for generating and evaluating competence portraits of compliance doctors

Abstract

The invention relates to the field of medical education management, and provides an AI-based competence portrait generation and evaluation method for a compliance training doctor, and the method comprises the steps: obtaining multi-source data covering the multi-dimensional competence of the compliance training doctor from a training management system, and introducing a metadata label system to label corresponding competence labels to construct a competence index set; and calling a large language model to analyze the ability index set based on a Dify platform, comparing the ability indexes of the current doctor in real time to trigger an intelligent early warning mechanism, and dynamically generating a doctor ability portrait and a comprehensive analysis report through the large language model based on the ability index set and / or an anomaly detection result. According to the method, efficient integration of multi-source data is realized, unified labeling and rapid calling of support capability indexes of a metadata label system are introduced, and capability portrait generation, compliance analysis and automatic output of personalized improvement suggestions are realized in combination with big language model analysis based on a Dify platform and an RAG enhancement mechanism; the problems of data dispersion, insufficient structuralization and intelligent analysis are effectively solved.

Description

Technical Field

[0001] The present invention relates to the field of medical education management, and in particular to a method and system for generating and evaluating the competence profile of resident physicians based on AI. Background Art

[0002] Standardized resident training is a crucial institutional arrangement for improving the quality of clinical talent. Currently, most hospitals have implemented resident training management systems to support the digital management of teaching activities, rotation arrangements, and evaluation and assessment. Existing systems are typically deployed on the Alibaba Cloud platform, using the Java development language and front-end and back-end frameworks, combined with traditional relational databases and application servers. These systems implement functional modules such as rotation plan management, electronic rotation manuals, 360-degree teaching evaluations, and final assessments, playing a positive role in basic data collection and process management.

[0003] However, with the rise of intelligent technologies such as artificial intelligence and big models, traditional residency training management systems have exposed a series of shortcomings, making it difficult to meet the current intelligent development needs of data-driven training quality improvement. They suffer from defects such as insufficient data dispersion and structuring, lack of dynamic analysis and intelligent feedback mechanisms, and limited technical scalability. For example: ① Although the existing system implements basic structured data storage, its field design is mainly oriented towards basic teaching management and information registration scenarios, and has not been structurally optimized for the needs of AI training and model inference. Key indicator data, such as department attendance rate, participation in skill operation teaching activities, skill operation scores, etc., are scattered across multiple heterogeneous table structures. There is a lack of unified field naming standards and metadata management mechanisms, resulting in inconsistent data semantics and low call efficiency. At the same time, the system contains a high proportion of unstructured data, such as department summaries, teaching evaluation texts, and case records, and the cleaning and parsing costs are high, further exacerbating the burden of data preprocessing and limiting the rapid deployment and effective operation of intelligent analysis modules.

[0004] ② The current system only performs static storage of unstructured text, making it difficult to perform deep semantic analysis such as semantic tag extraction and ability and behavior interpretation. It is also unable to model and evaluate key implicit characteristics such as trainees' work attitudes and skill proficiency, resulting in managers being unable to fully grasp the effectiveness of training.

[0005] ③ Most systems only provide static reports and chart displays, and cannot identify abnormal behaviors in real time based on multi-source data, such as continuous failure to meet assessment standards and large fluctuations in attendance. They need to rely on manual review and experience judgment to complete student risk identification and intervention, which is inefficient and highly subjective, affecting the scientific nature and timeliness of management.

[0006] ④ The current systems generally do not have embedded AI model capabilities, and lack the support capabilities for connecting to new intelligent components such as the large language model (LLM), RAG semantic enhancement mechanism, and multimodal data processing. They are difficult to meet expansion requirements such as intelligent capability assessment, personalized learning recommendations, and automatic report generation. The upgrade cost is high and the response cycle is long.

[0007] Therefore, there is an urgent need for an AI-based method and system for generating and evaluating the competence portraits of physicians in training, opening up channels for structured and unstructured data, integrating AI analysis and semantic modeling capabilities, building a portrait system with trainee capabilities as the core, and possessing intelligent evaluation capabilities such as risk identification, feedback suggestions, and automatic reporting, thereby realizing intelligent and precise training quality management. Summary of the Invention

[0008] In response to the problems of data dispersion, lack of structuring, and lack of intelligent analysis and feedback capabilities in the existing technology, the present invention proposes an AI-driven method and system for generating and intelligently evaluating the competence portraits of resident physician training students. By constructing a multi-source data fusion and integration mechanism, introducing a metadata tag system, configuring AI analysis workflows based on the Dify platform, and combining rule engines with large language models to achieve competence evaluation and dynamic report generation, it solves technical problems in training management such as low data retrieval efficiency, incomplete competence evaluation, lack of intelligent early warning and personalized improvement plans, and significantly improves the intelligence and refinement of training quality management.

[0009] The present invention provides an AI-based method for generating and evaluating the ability portrait of resident physicians, which is characterized by the following steps: Step S1: Obtain multi-source data covering the multi-dimensional capabilities of resident physicians from the training management system, and pre-process the multi-source data using ETL tools. Step S2: Perform multi-dimensional aggregate statistics on the capability indicators extracted from structured data, introduce a predefined metadata tag system to annotate the corresponding capability tags, and build a unified capability indicator set for AI analysis. Step S3: Based on the Dify platform, the large language model is called to conduct a comprehensive analysis of the ability indicator set. The threshold range of each ability indicator is dynamically calculated in combination with historical training data. The ability indicators of the current doctors are compared in real time. When it is detected that the ability indicators exceed the preset threshold range, the intelligent early warning mechanism is automatically triggered and the early warning information is pushed. Based on the ability indicator set and / or anomaly detection results, the large language model is used to dynamically generate a doctor's ability portrait and a comprehensive analysis report.

[0010] Furthermore, step S1 includes: In response to a preset timed task scheduler or an external task scheduler, the data processing service is started to automatically obtain multi-source data from the data interface or database of the training management system. The multi-source data includes structured data such as attendance data, rotation records, and assessment scores, as well as unstructured data such as departmental summaries, teaching evaluation texts, and medical records; Use ETL tools to preprocess multi-source data, including data cleaning, field standardization, and format unification.

[0011] Further, step S2 includes, Dynamically load the analysis service class and instantiate the corresponding structured data analysis service through the reflection mechanism based on the defined data object class; Calling the analysis service to perform multi-dimensional aggregate statistics on structured data, including but not limited to attendance rate, skill assessment scores, grade average, base and grade ranking, or any other ability indicator; The aggregated statistical results are encapsulated as capability indicator data records and written into the resident physician capability profile data table for subsequent AI analysis; A predefined metadata labeling system is introduced to label the fields of competency indicators with corresponding competency labels based on the field-competency mapping rules in the resident physician competency profile data table; Build a unified set of capability indicators, including structured aggregated data and its capability labels, for subsequent physician capability profile generation, report analysis, and early warning comparison.

[0012] Furthermore, step S2 also includes preprocessing the unstructured data using ETL tools and adding corresponding source data tags for subsequent retrieval and analysis.

[0013] Furthermore, in step S3, the large language model is called based on the Dify platform to conduct a comprehensive analysis of the capability indicator set, including: Based on the Dify platform, a predefined workflow is triggered, and the large language model is called through the configured Prompt template. The capability indicator set is used as input data, and the local knowledge base is loaded for multi-dimensional context fusion analysis. The local knowledge base is the knowledge content stored in the vector database of training outlines, assessment standards, and policy documents.

[0014] Furthermore, in step S3, the threshold range of each ability indicator is dynamically calculated in combination with historical training data, and the ability indicator of the current doctor is compared in real time. When it is detected that the ability indicator exceeds the preset threshold range, the intelligent early warning mechanism is automatically triggered and the early warning information is pushed. Based on the ability indicator set and / or abnormal detection results, the doctor's ability portrait and comprehensive analysis report are dynamically generated through the large language model, including: Based on historical training data, the standard fluctuation range of each capability indicator is statistically calculated, and the threshold range of each indicator is dynamically generated and stored in the threshold management table or cache; Obtain the capability indicator data from the capability indicator set of the resident physicians in real time and compare it with the threshold range; determine whether the capability indicator data exceeds the threshold range. If it exceeds the threshold range, mark the abnormality and record the abnormality reason and indicator information; The current doctor's ability indicator set, annotated ability labels and abnormal identification are used as input and passed to the LLM node of the workflow. Through the configured Prompt template, the large language model is called to generate a physician ability profile and report analysis that includes performance analysis of the regulated physician's ability, standard comparison analysis, strengths and weaknesses analysis, and improvement suggestions.

[0015] Furthermore, step S3 further includes: When the output results confirm that there are abnormalities or shortcomings in the ability indicators, the workflow calls the preset push interface to push early warning information to managers and teaching teachers.

[0016] Based on the same inventive concept, the present invention also provides an AI-based system for generating and evaluating the ability portrait of resident physicians, which adopts the above-mentioned method for generating and evaluating the ability portrait of resident physicians, including: The data collection module obtains multi-source data covering the multi-dimensional capabilities of regulated physicians from the training management system and pre-processes the multi-source data using ETL tools. The data processing module performs multi-dimensional aggregate statistics on capability indicators extracted from structured data, introduces a predefined metadata tag system to annotate corresponding capability tags, and builds a unified capability indicator set for AI analysis. The data analysis module calls the large language model based on the Dify platform to conduct a comprehensive analysis of the ability indicator set, dynamically calculates the threshold range of each ability indicator based on historical training data, and compares the current doctor's ability indicators in real time. When it is detected that the ability indicator exceeds the preset threshold range, the intelligent early warning mechanism is automatically triggered and the early warning information is pushed. Based on the ability indicator set and / or anomaly detection results, the large language model is used to dynamically generate a doctor's ability portrait and comprehensive analysis report.

[0017] Furthermore, the data processing module includes: The service loading unit is used to dynamically load the analysis service class and instantiate the corresponding structured data analysis service through the reflection mechanism based on the defined data object class; Aggregate statistics unit, used to call the analysis service to perform multi-dimensional aggregate statistics on structured data, including but not limited to attendance rate, skill assessment score, grade average, base and grade ranking, or any other ability indicator; The indicator record generation unit is used to encapsulate the aggregated statistical results into capability indicator data records and write them into the resident physician capability profile data table; introduce a predefined metadata label system, and annotate the fields of the capability indicators with corresponding capability labels based on the field-capability mapping rules in the resident physician capability profile data table; build a unified capability indicator set, which includes structured aggregated data and its capability labels, for physician capability profile generation, report analysis and early warning comparison.

[0018] Furthermore, the data analysis module includes, The configuration unit is used to trigger predefined workflows based on the Dify platform. It calls the large language model through the configured prompt template, takes the capability indicator set as input data, and loads the local knowledge base for multi-dimensional context fusion analysis. The local knowledge base is the knowledge content stored in the vector database of training outlines, assessment standards, and policy documents.

[0019] The anomaly detection unit is used to calculate the standard fluctuation range of each ability indicator based on historical training data, dynamically generate the threshold range of each indicator, and store it in the threshold management table or cache; obtain the ability indicator data of the ability indicator set of regular training doctors in real time, and compare it with the threshold range; determine whether the ability indicator data exceeds the threshold range. If there is an excess, mark the anomaly and record the cause of the anomaly and indicator information; The analysis unit is used to input the current doctor's ability indicator set, annotated ability labels and abnormal identification into the LLM node of the workflow. Through the configured prompt template, it calls the large language model to generate a physician ability profile and report analysis that includes performance analysis of the regulated physician's ability, standard comparison analysis, strengths and weaknesses analysis, and improvement suggestions.

[0020] Compared with the prior art, the present invention has at least one of the following technical effects: The present invention constructs an AI-driven resident physician training student competency profile generation and intelligent assessment system, achieving efficient integration and standardized management of multi-source data, including structured and unstructured data. It innovatively introduces a metadata tag system to support unified labeling and rapid call of competency indicators. Combined with the large language model analysis and RAG enhancement mechanism based on the Dify platform, it realizes the automatic output of competency profile generation, compliance analysis and personalized improvement suggestions, effectively solving the problems of data dispersion, insufficient structuring, and lack of intelligent analysis and feedback capabilities in the existing technology, and significantly improving the intelligence, refinement and real-time response capabilities of training management.

[0021] (1) This invention realizes the integrated storage and management of structured and unstructured data by introducing a metadata tag system and a unified capability indicator set, solving the problems of data dispersion, semantic inconsistency, and low retrieval efficiency in the existing technology, and improving the efficiency and consistency of data retrieval. (2) The present invention configures AI analysis workflow based on the Dify platform, combines the rule engine and large language model, and realizes multi-dimensional dynamic analysis of resident physician capabilities through RAG enhancement mechanism and local knowledge base support. It can generate personalized capability portraits and targeted improvement suggestions, significantly improving the intelligence and refinement of the training process.

[0022] (3) The present invention adopts a dynamic threshold calculation algorithm, which can monitor the performance of residents in real time based on historical training data, automatically identify risk behaviors such as abnormal attendance and declining grades, and intelligently trigger early warnings, push early warning information and personalized improvement plans, thereby improving the scientific nature and timeliness of training management.

[0023] (4) The present invention adopts a microservice architecture and low-code platform design, which can flexibly expand capability assessment dimensions such as scientific research capabilities, communication and collaboration capabilities, etc., support rapid deployment and cross-system integration, reduce technology upgrade and maintenance costs, and facilitate promotion and application in different medical institutions. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments: Figure 1 This is a flowchart of the steps of the method for generating and evaluating the ability portrait of resident physicians based on AI in the present invention. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application. The specific implementation methods of the present invention are described below in conjunction with the drawings and embodiments.

[0026] First embodiment Existing standardized resident training management systems are widely deployed in hospitals at all levels, assuming core functions such as resident rotation management, teaching activity records, assessment and evaluation, and electronic rotation manuals. However, through in-depth research and technical verification in long-term resident training practices, the inventors have discovered that such systems have the following key technical bottlenecks when supporting advanced applications such as capability profiling and intelligent assessment: First, data is fragmented and semantically inconsistent. In existing systems, key competency-related data, such as attendance rates, skill assessments, and teaching activity participation records, is stored in separate business tables or modules. Field definitions lack a unified competency label or metadata identification mechanism. This requires repeated cleaning and mapping of various data types during model training and inference, severely impacting data access efficiency and model effectiveness, making it difficult to support the rapid integration of large language models or intelligent engines.

[0027] Secondly, there is a lack of in-depth utilization of unstructured data. For example, although there is a rich accumulation of text data such as department summaries, teaching evaluations, and case records, traditional systems only store them statically and have not established a semantic extraction, label annotation, or vectorized indexing system based on capability labels. This results in the inability of large language models to effectively call upon the context of unstructured data, which restricts the integrity and evaluation depth of capability portraits.

[0028] Finally, traditional systems are mainly oriented towards static statistics and manual reporting, and do not integrate dynamic threshold algorithms, contextual comparison mechanisms or intelligent early warning engines. They are unable to achieve automatic detection, early warning and generation of personalized intervention recommendations for capability risks based on the fusion of historical data and real-time data. They rely heavily on manual screening and experience-based judgment, which can easily lead to delays and deviations.

[0029] To address these technical shortcomings, the inventors, after in-depth technical analysis and solution demonstration, creatively proposed an AI-driven system and method for generating and intelligently evaluating the competency profiles of resident physician trainees. Key breakthroughs of this solution include: (1) Introducing a metadata tag-driven data integration framework. By introducing a unified metadata capability tag system for structured and unstructured multi-source data fields, unified semantic annotation, capability dimension classification and intelligent call of data can be achieved, providing a highly consistent capability indicator set for AI analysis.

[0030] (2) Build a multi-dimensional AI workflow based on the Dify low-code platform, integrating the rule engine, large language model (LLM), and local vector knowledge base (supporting RAG enhancement mechanism), and automatically complete capability data analysis, dynamic capability portrait generation, standard comparison analysis, and personalized improvement suggestion output.

[0031] (3) Design dynamic thresholds and intelligent early warning mechanisms to automatically calculate the standard fluctuation range of each ability indicator based on historical training data, realize real-time monitoring of ability performance, automatic identification of abnormal indicators, early warning push, and generation of personalized intervention task lists.

[0032] Therefore, a customized database and intelligent model are used to implement competency assessment and early warning. A new format is used to store multi-dimensional data, including rotation plans, teaching activities, electronic rotation manuals, 360-degree evaluations, and final assessments. Metadata tags are designed to support rapid retrieval, and a microservices architecture is integrated to achieve real-time synchronization and updates, ensuring data currency. The system also builds a competency profile generation engine, integrating modules such as behavioral analysis and competency assessment. For example, it calculates a work motivation index based on attendance rate and leave type. A skill heat map is generated by combining the electronic rotation manual and final assessments. Cluster analysis of assessment results is performed to identify weak links. Profile data is regularly updated using an AI model and synchronized to the management system front-end. A real-time early warning mechanism is implemented in the system. When the work motivation index or skill score is abnormal, an alert is automatically sent to the training office and simulated training recommendations are provided. Based on the profile, personalized learning plans are generated, and assessment complexity is dynamically adjusted, implementing an intelligent assessment model where "achieving standards leads to upgrading."

[0033] For example, in system applications, the data integration and AI workflow designed by the inventors can synchronize training data in real time after trainees submit their rotation manuals. Through AI models, the gap between their attendance rate and skill scores and the syllabus standards can be dynamically analyzed. A capability radar chart and improvement suggestions can be automatically generated. In the event of an abnormality, these suggestions can be immediately pushed to the training office and instructors, significantly improving the intelligence and refinement of training management. The specific implementation methods are as follows: The present invention provides an AI-based method for generating and evaluating the ability portrait of resident physicians, which is characterized by the following steps: Step S1: Obtain multi-source data covering the multi-dimensional capabilities of resident physicians from the training management system, and pre-process the multi-source data using ETL tools. Step S2: Perform multi-dimensional aggregate statistics on the capability indicators extracted from structured data, introduce a predefined metadata tag system to annotate the corresponding capability tags, and build a unified capability indicator set for AI analysis. Step S3: Based on the Dify platform, the large language model is called to conduct a comprehensive analysis of the ability indicator set. The threshold range of each ability indicator is dynamically calculated in combination with historical training data. The ability indicators of the current doctors are compared in real time. When it is detected that the ability indicators exceed the preset threshold range, the intelligent early warning mechanism is automatically triggered and the early warning information is pushed. Based on the ability indicator set and / or anomaly detection results, the large language model is used to dynamically generate a doctor's ability portrait and a comprehensive analysis report.

[0034] Furthermore, step S1 includes: In response to a preset timed task scheduler or an external task scheduler, the data processing service is started to automatically obtain multi-source data from the data interface or database of the training management system. The multi-source data includes structured data such as attendance data, rotation records, and assessment scores, as well as unstructured data such as departmental summaries, teaching evaluation texts, and medical records; Use ETL tools to preprocess multi-source data, including data cleaning, field standardization, and format unification.

[0035] Specifically, the data sources include: Structured data: attendance rate, assessment results, evaluation scores, and activity participation, from the existing resident physician standardized training platform database, such as MySQL / Oracle.

[0036] Unstructured data: student exit summaries, teaching evaluation texts, and case records, which come from the interface of the existing resident physician standardized training platform.

[0037] Knowledge base: training outlines, assessment standards, and policy documents, used to build a local vector database, such as Milvus.

[0038] Data preprocessing ETL process: Use Dify's built-in DatasetETL tool to clean data and handle missing values ​​and outliers.

[0039] Standardization: Unify field formats (such as date and rating level) to ensure data consistency.

[0040] Further, step S2 includes, Dynamically load the analysis service class and instantiate the corresponding structured data analysis service through the reflection mechanism based on the defined data object class; Calling the analysis service to perform multi-dimensional aggregate statistics on structured data, including but not limited to attendance rate, skill assessment scores, grade average, base and grade ranking, or any other ability indicator; The aggregated statistical results are encapsulated as capability indicator data records and written into the resident physician capability profile data table for subsequent AI analysis; A predefined metadata labeling system is introduced to label the fields of competency indicators with corresponding competency labels based on the field-competency mapping rules in the resident physician competency profile data table; Build a unified set of capability indicators, including structured aggregated data and its capability labels, for subsequent physician capability profile generation, report analysis, and early warning comparison.

[0041] Furthermore, step S2 also includes preprocessing the unstructured data using ETL tools and adding corresponding source data tags for subsequent retrieval and analysis.

[0042] Furthermore, in step S3, the large language model is called based on the Dify platform to conduct a comprehensive analysis of the capability indicator set, including: Based on the Dify platform, a predefined workflow is triggered, and the large language model is called through the configured Prompt template. The capability indicator set is used as input data, and the local knowledge base is loaded for multi-dimensional context fusion analysis. The local knowledge base is the knowledge content stored in the vector database of training outlines, assessment standards, and policy documents.

[0043] Specifically, the core workflow of the Dify platform includes data input, data parsing, multi-dimensional analysis, student portrait generation, report generation, and visual output. Node configuration: Data parsing node: parses raw data in Excel / JSON format and extracts key fields such as attendance rate and assessment scores.

[0044] The analysis nodes include, Rule engine: Based on preset thresholds, such as attendance rate <80%, triggers an early warning to generate preliminary analysis.

[0045] LLM analysis: Call the local knowledge base and generate a comprehensive evaluation through the prompt template, such as "Student A performed well in case analysis, but his skills need to be improved."

[0046] Furthermore, in step S3, the threshold range of each ability indicator is dynamically calculated in combination with historical training data, and the ability indicator of the current doctor is compared in real time. When it is detected that the ability indicator exceeds the preset threshold range, the intelligent early warning mechanism is automatically triggered and the early warning information is pushed. Based on the ability indicator set and / or abnormal detection results, the doctor's ability portrait and comprehensive analysis report are dynamically generated through the large language model, including: Based on historical training data, the standard fluctuation range of each capability indicator is statistically calculated, and the threshold range of each indicator is dynamically generated and stored in the threshold management table or cache; Obtain the capability indicator data from the capability indicator set of the resident physicians in real time and compare it with the threshold range; determine whether the capability indicator data exceeds the threshold range. If it exceeds the threshold range, mark the abnormality and record the abnormality reason and indicator information; The current doctor's ability indicator set, annotated ability labels and abnormal identification are used as input and passed to the LLM node of the workflow. Through the configured Prompt template, the large language model is called to generate a physician ability profile and report analysis that includes performance analysis of the regulated physician's ability, standard comparison analysis, strengths and weaknesses analysis, and improvement suggestions.

[0047] (1) The student portrait is constructed as follows: Portrait Dimension Ability dimensions: clinical skills, theoretical level, and teaching feedback scores.

[0048] Behavioral dimensions: attendance patterns, study time, and case participation.

[0049] Risk warning: low attendance rate, continuous failure to meet assessment standards, etc.

[0050] LLM prompt word design Generate student profiles based on the following data: Attendance rate: 95% Theoretical exam: 85 / 100 Skill assessment: 78 / 100 Teaching evaluation: Excellent communication skills, but operational standardization needs to be improved Requirements: Summarize the advantages and disadvantages in points, put forward suggestions for improvement, and use concise language.

[0051] Knowledge base enhancement: Link training outlines to match the gap between trainees’ abilities and outline requirements.

[0052] (2) Contents of the training process analysis report: Report Contents Personal report: radar chart of student capabilities, timeline of key events such as completion assessment time, and suggestions for improvement.

[0053] Group report: department training quality ranking, analysis of common weaknesses, such as "the surgical team's suturing skills compliance rate is only 60%."

[0054] Automatic generation Template engine: Preset report templates and fill in dynamic data through Dify's code execution nodes.

[0055] Visualization component: Integrate ECharts / line chart to display learning curve and comparative analysis.

[0056] Furthermore, step S3 further includes: When the output results confirm that there are abnormalities or shortcomings in the ability indicators, the workflow calls the preset push interface to push early warning information to managers and teaching teachers.

[0057] Specifically, system integration and deployment include: (1) Local deployment: Dify containerization: Deploy through Docker, configure PostgreSQL database and Redis cache.

[0058] Permission control: Based on the RBAC model, it distinguishes between administrators (viewing data of the entire hospital), teaching secretaries and teaching teachers (viewing their departments), and students (only personal data).

[0059] (2) Interface extension: Data synchronization: Synchronize existing system data through scheduled tasks.

[0060] Third-party tools: Integrate DingTalk / WeChat for Business to push warning notifications and reports.

[0061] Second embodiment Based on the same inventive concept, the present invention also provides an AI-based system for generating and evaluating the ability portraits of resident physicians. It adopts the above-mentioned method for generating and evaluating the ability portraits of resident physicians, and realizes the dynamic generation of the ability portraits of resident physicians and problem warning through customized database design, multi-dimensional data integration and large model empowerment.

[0062] 1. System architecture and data integration The system is deployed on the Alibaba Cloud platform, using the Java development language, back-end frameworks, and front-end frameworks, combined with traditional relational databases and application servers. Its core innovations lie in: optimized database design and structured storage solutions: integrating scattered fields in the regular training system (such as rotation plans, attendance records, electronic rotation manuals, departmental assessments, and 360-degree evaluations), and supporting rapid invocation of AI models by defining unified metadata tags (such as "proficiency" to indicate skill mastery). Incremental update mechanism: utilizing a microservice architecture to achieve real-time data synchronization. For example, each time a physician submits disease content in the electronic rotation manual, the system automatically triggers a field update and accelerates queries through caching.

[0063] 2. Resident Physician Competency Profile Generation Engine The system uses the Ali Qianwen (Qwen) 32B large model and the Dify platform to build dynamic capability portraits and multi-dimensional data integration. Behavior analysis module: Attendance analysis calculates the "work enthusiasm index". If the index is lower than 0.6 for two consecutive weeks, an early warning is triggered. Skill assessment module: The skill mastery calculation is based on the data reported in the electronic rotation manual, combined with the skill requirements of the third-year physicians in the corresponding departments in the "Standardized Training Content and Standards for Resident Physicians", and the RAG retrieval technology is used to compare the gaps and generate feedback. Agent automation process: The Agent workflow is orchestrated through the Dify platform to perform the following steps regularly: 1. Extract the latest data from MySQL and convert it into a special format for AI databases; 2. The Ali Qianwen model calls RAG to retrieve the "Standardized Training Content and Standards for Resident Physicians" to generate a personalized capability report; 3. Synchronize the results to the front end, and the training office can view the portrait details. The specific implementation steps include, The data collection module obtains multi-source data covering the multi-dimensional capabilities of regulated physicians from the training management system and pre-processes the multi-source data using ETL tools. The data processing module performs multi-dimensional aggregate statistics on capability indicators extracted from structured data, introduces a predefined metadata tag system to annotate corresponding capability tags, and builds a unified capability indicator set for AI analysis. The data analysis module calls the large language model based on the Dify platform to conduct a comprehensive analysis of the ability indicator set, dynamically calculates the threshold range of each ability indicator based on historical training data, and compares the current doctor's ability indicators in real time. When it is detected that the ability indicator exceeds the preset threshold range, the intelligent early warning mechanism is automatically triggered and the early warning information is pushed. Based on the ability indicator set and / or anomaly detection results, the large language model is used to dynamically generate a doctor's ability portrait and comprehensive analysis report.

[0064] Furthermore, the data processing module includes: The service loading unit is used to dynamically load the analysis service class and instantiate the corresponding structured data analysis service through the reflection mechanism based on the defined data object class; Aggregate statistics unit, used to call the analysis service to perform multi-dimensional aggregate statistics on structured data, including but not limited to attendance rate, skill assessment score, grade average, base and grade ranking, or any other ability indicator; The indicator record generation unit is used to encapsulate the aggregated statistical results into capability indicator data records and write them into the resident physician capability profile data table; introduce a predefined metadata label system, and annotate the fields of the capability indicators with corresponding capability labels based on the field-capability mapping rules in the resident physician capability profile data table; build a unified capability indicator set, which includes structured aggregated data and its capability labels, for physician capability profile generation, report analysis and early warning comparison.

[0065] Furthermore, the data analysis module includes, The configuration unit is used to trigger predefined workflows based on the Dify platform. It calls the large language model through the configured prompt template, takes the capability indicator set as input data, and loads the local knowledge base for multi-dimensional context fusion analysis. The local knowledge base is the knowledge content stored in the vector database of training outlines, assessment standards, and policy documents.

[0066] The anomaly detection unit is used to calculate the standard fluctuation range of each ability indicator based on historical training data, dynamically generate the threshold range of each indicator, and store it in the threshold management table or cache; obtain the ability indicator data of the ability indicator set of regular training doctors in real time, and compare it with the threshold range; determine whether the ability indicator data exceeds the threshold range. If there is an excess, mark the anomaly and record the cause of the anomaly and indicator information; The analysis unit is used to input the current doctor's ability indicator set, annotated ability labels and abnormal identification into the LLM node of the workflow. Through the configured prompt template, it calls the large language model to generate a physician ability profile and report analysis that includes performance analysis of the regulated physician's ability, standard comparison analysis, strengths and weaknesses analysis, and improvement suggestions.

[0067] Third embodiment In order to enable those skilled in the art to have a more comprehensive and in-depth understanding of the technical solution of the present invention and its innovative features and technical advantages over the prior art, the key technical means, innovative design and improved contents of the present invention that are different from the prior art are now described in detail in conjunction with the third embodiment. This embodiment is mainly aimed at the actual application scenarios of ability portrait generation and intelligent assessment in standardized training of resident physicians, focusing on demonstrating the inventor's technological breakthroughs and differentiated designs in data integration mechanisms, metadata labeling systems, large language model integration and intelligent early warning mechanisms, so as to clearly reflect the creative conception of the present invention and its technical effects in practical applications.

[0068] (1) In traditional resident physician training management systems, attendance data, assessment scores, rotation records, and other information are usually stored in different business module tables. For example, attendance information is stored in the attendance table (ATTENDANCE_TABLE), assessment scores are stored in the score table (EXAM_RESULT_TABLE), and rotation records are stored in the rotation table (ROTATION_TABLE). Due to the inconsistent naming of the fields in each table and the lack of standardized semantic specifications, developers need to write complex SQL scripts to manually splice data during data analysis and competency assessment, which is time-consuming and error-prone. At the same time, the meaning of the fields is unclear. For example, the score field may represent the theoretical test score or the skill operation score, which further increases the difficulty of data interpretation and is prone to misjudgment or omission. In addition, the existing system lacks a unified concept of competency indicators, and the efficiency of data aggregation and analysis is low, making it difficult to support large-scale, intelligent data retrieval and processing.

[0069] In response to the above problems, the present invention innovatively introduces a metadata labeling system and a unified capability indicator set. The inventors designed a set of standardized capability label mapping rules, for example, uniformly labeling the ATTENDANCE_RATE and ATT_RATE fields as "work enthusiasm", labeling SKILL_SCORE as "skill mastery", and labeling ROTATION_PARTICIPATION as "clinical rotation participation". During data processing, the system automatically completes labeling based on the field and capability mapping relationship, and writes the labeled data into the capability portrait data table to achieve the integrated management of structured data and unstructured data. For unstructured data such as department summaries and teaching evaluations, the present invention parses the text through a large language model, extracts semantic information and maps it to corresponding capability labels such as "communication and collaboration skills" and "scientific research awareness", and uniformly writes it into the capability indicator set.

[0070] With this design, the system can retrieve and analyze competency data quickly and accurately, eliminating the need to manually join business tables or interpret field definitions. This significantly improves the efficiency and consistency of data retrieval. For example, a teaching secretary might query for a list of students with a skill mastery level < 60%. Traditional systems require manual linking of multiple tables, field interpretation, and query combination. However, this new system only requires searching based on the competency tag "skill mastery." The system automatically returns a list of eligible students, along with rich data support such as their grade rankings and fluctuation trends.

[0071] (2) Similarly, existing resident training management systems generally only support basic static data display functions, such as displaying attendance in the form of bar charts and presenting assessment results in the form of lists. If training managers need to conduct a comprehensive assessment of trainees' abilities, they often need to manually compare various data with the requirements of the training syllabus and manually write evaluation opinions and improvement suggestions. This is not only labor-intensive and time-consuming, but also prone to evaluation bias due to subjective judgment. At the same time, traditional systems lack the ability to automatically identify shortcomings in abilities and are unable to generate personalized improvement plans based on data in real time.

[0072] To address the aforementioned issues, the present invention designs a configurable AI analysis workflow based on the Dify platform, enabling the automatic generation of competency profiles and the intelligent delivery of personalized improvement plans. Within the system, when a workflow is triggered, such as a scheduled task or assessment data update, the system takes a set of competency indicators, including structured data and unstructured competency data parsed by a large language model (LLM), as input and invokes the large language model (LLM). The LLM automatically loads a local vector knowledge base encompassing training syllabi, assessment standards, and policy documents, and dynamically compares and analyzes the competency data context using the RAG enhancement mechanism. This process generates a multi-dimensional competency profile of residents meeting training requirements. This profile includes the resident's performance in clinical skills, theoretical knowledge, and scientific research awareness, among other dimensions. A comparative analysis with the training standards is performed, e.g., whether skill mastery meets standards and whether theoretical scores deviate from these standards. Based on this, personalized improvement recommendations are generated, such as recommendations for simulated operation training or participation in scientific research projects. Finally, the system automatically generates a preliminary evaluation report and delivers it to training managers and instructors via the management platform or channels such as WeChat for Business and DingTalk.

[0073] For example, when a trainee's clinical skills score is 75 points but there is no record of participation in scientific research projects, the traditional system requires managers to manually summarize the data and write evaluation opinions. However, the present invention can automatically output the conclusion through AI: "Student A has good clinical skills, but insufficient participation in scientific research. It is recommended to arrange for a scientific research mentor to guide and participate in the research project", significantly improving the intelligence and work efficiency of training management.

[0074] (3) The existing resident physician training management system mainly relies on manual operations to identify risky behaviors. For example, training managers need to export reports every month and manually screen trainees' attendance records, assessment scores, and other data to identify risky behaviors such as abnormal attendance and declining grades. This method not only has a long cycle (often taking 1-2 weeks or even longer), but is also prone to omissions and false alarms due to large amounts of data and subjective omissions. At the same time, the existing system lacks differentiated early warning capabilities based on the characteristics of different departments or grades, and cannot dynamically adjust thresholds (for example, the attendance requirements for surgical departments are higher than those for internal medicine departments). The early warning is not targeted and scientific enough.

[0075] To overcome the above-mentioned shortcomings, the present invention innovatively designs a dynamic threshold algorithm and an intelligent early warning mechanism. Based on historical training data, the system automatically calculates the fluctuation range of the ability indicators of different departments, grades, and stages to form a dynamic threshold range (for example, the attendance threshold of surgical residents is set at 90%, while that of internal medicine residents can be set at 80%). In daily operation, the system collects and compares the current trainee ability indicator data (such as attendance rate, skill assessment scores, rotation participation, etc.) in real time. Once the attendance rate is lower than the dynamic threshold for two consecutive weeks, or the skill assessment scores drop by more than the early warning threshold compared to the previous stage, the system automatically generates an abnormality mark and records the cause of the abnormality and the specific ability indicator.

[0076] At the same time, the system immediately triggers an intelligent early warning mechanism, pushing warning information in real time to instructors and training management departments via integrated platforms like WeChat and DingTalk. Warning information includes personalized improvement suggestions (such as suggesting rescheduling, participating in simulated operation training, and assigning dedicated personnel for follow-up guidance). It also automatically generates a personalized task list to assist managers in quickly implementing interventions.

[0077] For example, when a trainee's attendance rate is less than 80% for two consecutive weeks and their skill assessment scores drop by 10% month-on-month, the traditional system may need to wait until the monthly report is released for manual identification and notification. However, the system of the present invention can detect in real time after the data fails to meet the standards, immediately push a DingTalk notification, and attach a generated simulation training task list, effectively improving the response speed and accuracy of training management.

[0078] (4) Existing resident physician training management systems have significant limitations when expanding capability dimensions (e.g., scientific research capability, communication and collaboration capability, etc.). When new evaluation indicators or dimensions are needed, it is often necessary to make large-scale modifications to the existing database table structure and redesign the data table fields, front-end report templates, and interface services. This results in a long development cycle, high costs, and high risks. In addition, the integration of such systems with other hospital business systems such as scientific research and performance management is complex, data flow is poor, and information silos are easily formed, which restricts the overall effectiveness of training management.

[0079] To address these issues, the present invention utilizes a microservices architecture and low-code technology to design a capability profiling and assessment system. Within the system, each capability dimension is flexibly defined through a metadata tagging system and mapping relationships to capability indicators. Expanding a new dimension (e.g., scientific research capability) requires only adding the relevant tags and mapping rules to the capability indicator mapping table, without modifying the main database structure. Furthermore, the Dify platform's low-code features allow for direct configuration of workflow nodes, enabling rapid collection, analysis, and report presentation of new capability data without extensive coding. This solution significantly shortens the development cycle, enabling the entire process from definition to launch of a new capability dimension to be completed within a week.

[0080] In addition, the system of the present invention supports cross-system integration through standardized API interfaces. For example, it can quickly access external platforms such as scientific research management systems and performance management systems to automatically acquire and analyze data such as scientific research results and paper publications, thereby forming a complete capability portrait in the dimension of scientific research capabilities.

[0081] For example, when a hospital requests the addition of a new research capability indicator to its existing capacity assessment, traditional systems often require one to three months to complete database expansion, front-end refactoring, and interface modifications. However, the system presented in this invention only requires configuring the new "research capability" metadata tag, expanding workflow nodes, and integrating with the research system interface. This allows for a launch within a week, significantly improving response efficiency and reducing development and maintenance costs.

[0082] The scope of protection of the present invention is not limited to the above-mentioned embodiments. All technical solutions based on the concept of the present invention are within the scope of protection of the present invention. It should be noted that for those skilled in the art, various improvements and modifications that do not depart from the principles of the present invention should also be considered within the scope of protection of the present invention.

[0083] The technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0084] It should be noted that the above embodiments can be freely combined as needed. The above description is only a preferred embodiment of the present invention. It should be pointed out that those skilled in the art can make several improvements and modifications without departing from the principles of the present invention, and such improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for generating and evaluating the ability portrait of resident physicians based on AI, characterized in that: The steps include, Step S1: Obtain multi-source data covering the multi-dimensional capabilities of resident physicians from the training management system, and pre-process the multi-source data using ETL tools. Step S2: Perform multi-dimensional aggregate statistics on the capability indicators extracted from the structured data, introduce a predefined metadata tag system to annotate the corresponding capability tags, and build a unified capability indicator set for AI analysis. Step S3: Based on the Dify platform, the large language model is called to conduct a comprehensive analysis of the capability indicator set, and the threshold range of each capability indicator is dynamically calculated in combination with historical training data. The capability indicators of the current doctor are compared in real time. When it is detected that the capability indicator exceeds the preset threshold range, the intelligent early warning mechanism is automatically triggered and the early warning information is pushed. Based on the capability indicator set and / or the abnormality detection results, the doctor's capability portrait and comprehensive analysis report are dynamically generated through the large language model.

2. The method for generating and evaluating the ability portrait of a resident physician according to claim 1 is characterized in that: Step S1 includes, In response to a preset timed task scheduler or an external task scheduler, a data processing service is started to automatically obtain the multi-source data from the data interface or database of the training management system, wherein the multi-source data includes structured data such as attendance data, rotation records, and assessment scores, and unstructured data such as departmental summaries, teaching evaluation texts, and medical records; The ETL tool is used to perform data preprocessing on the multi-source data, and the data preprocessing includes data cleaning, field standardization and format unification processing.

3. The method for generating and evaluating the ability portrait of a resident physician according to claim 2 is characterized in that: Step S2 includes, Dynamically load the analysis service class and instantiate the corresponding structured data analysis service through the reflection mechanism based on the defined data object class; Calling the analysis service to perform multi-dimensional aggregate statistics on the structured data, including but not limited to attendance rate, skill assessment score, grade average, base and grade ranking, and any one of the ability indicators; The aggregated statistical results are encapsulated as capability indicator data records and written into the resident physician capability profile data table for subsequent AI analysis; Introducing the predefined metadata labeling system, and labeling the fields of the competency indicators with corresponding competency labels based on the field-competency mapping rules in the resident physician competency profile data table; Build a unified set of capability indicators, including structured aggregated data and its capability labels, for subsequent physician capability profile generation, report analysis and early warning comparison.

4. The method for generating and evaluating the ability portrait of a resident physician according to claim 3 is characterized in that: The step S2 further includes performing data preprocessing on the unstructured data by using the ETL tool and adding corresponding source data tags for subsequent retrieval and analysis.

5. The method for generating and evaluating the ability portrait of a resident physician according to claim 4 is characterized in that: In step S3, the large language model is called based on the Dify platform to conduct a comprehensive analysis of the capability indicator set, including: Based on the Dify platform, a predefined workflow is triggered, the large language model is called through the configured Prompt template, the capability indicator set is used as input data, and the local knowledge base is loaded for multi-dimensional context fusion analysis. The local knowledge base is the knowledge content stored in the vector database of training outlines, assessment standards, and policy documents.

6. The method for generating and evaluating the ability profile of a resident physician according to claim 5, characterized in that: In step S3, the threshold range of each ability indicator is dynamically calculated in combination with historical training data, and the ability indicator of the current doctor is compared in real time. When it is detected that the ability indicator exceeds the preset threshold range, the intelligent early warning mechanism is automatically triggered and the early warning information is pushed. Based on the ability indicator set and / or abnormal detection results, the doctor's ability portrait and comprehensive analysis report are dynamically generated through the large language model. include, Based on historical training data, the standard fluctuation range of each capability indicator is statistically calculated, the threshold range of each indicator is dynamically generated, and stored in a threshold management table or cache; Acquire the capability indicator data of the capability indicator set of the resident physician in real time and compare it with the threshold range; determine whether the capability indicator data exceeds the threshold range; if so, mark the abnormality and record the abnormality cause and indicator information; The capability indicator set of the current doctor, the marked capability label and the abnormal identification are taken as input and passed into the LLM node of the workflow. Through the configured Prompt template, the large language model is called to generate the physician capability portrait and the report analysis of the training doctor's capability performance analysis, standard comparison analysis, strengths and weaknesses analysis and improvement suggestions.

7. The method for generating and evaluating the ability profile of a resident physician according to claim 6, characterized in that: The step S3 further includes: When the output results confirm that there are abnormalities or shortcomings in the ability indicators, the workflow calls the preset push interface to push warning information to the manager and teaching teacher.

8. A system for generating and evaluating the ability profile of resident physicians based on AI, using the method for generating and evaluating the ability profile of resident physicians according to any one of claims 1 to 7, characterized in that: The data acquisition module obtains multi-source data covering the multi-dimensional capabilities of regulated physicians from the training management system, and pre-processes the multi-source data through ETL tools. The data processing module performs multi-dimensional aggregation statistics on the capability indicators extracted from the structured data, introduces a predefined metadata tag system to annotate the corresponding capability tags, and constructs a unified capability indicator set for AI analysis. The data analysis module calls the large language model based on the Dify platform, conducts a comprehensive analysis of the capability indicator set, dynamically calculates the threshold range of each capability indicator in combination with historical training data, and compares the capability indicators of the current doctor in real time. When it is detected that the capability indicator exceeds the preset threshold range, the intelligent early warning mechanism is automatically triggered and the early warning information is pushed. Based on the capability indicator set and / or abnormal detection results, the large language model is used to dynamically generate a doctor's capability portrait and a comprehensive analysis report.

9. The method for generating and evaluating the ability profile of a resident physician according to claim 8, characterized in that: Data processing module, including, The service loading unit is used to dynamically load the analysis service class and instantiate the corresponding structured data analysis service through the reflection mechanism based on the defined data object class; An aggregation statistics unit, configured to call the analysis service to perform multi-dimensional aggregation statistics on the structured data, including but not limited to attendance rate, skill assessment score, grade average, base and grade ranking, and any one of the ability indicators; An indicator record generation unit is used to encapsulate the aggregated statistical results into capability indicator data records and write them into the resident physician capability portrait data table; introduce the predefined metadata label system, and label the fields of the capability indicators with corresponding capability labels based on the field-capability mapping rules in the resident physician capability portrait data table; and construct a unified capability indicator set, which includes structured aggregated data and its capability labels, for use in the physician capability portrait generation, report analysis, and warning comparison.

10. The method for generating and evaluating the ability profile of a resident physician according to claim 9, characterized in that: Data analysis modules, including: a configuration unit configured to trigger a predefined workflow based on the Dify platform, invoke the large language model through a configured prompt template, use the capability indicator set as input data, and simultaneously load the local knowledge base for multi-dimensional context fusion analysis, where the local knowledge base is knowledge content stored in a vector database of training outlines, assessment standards, and policy documents; An anomaly detection unit, configured to calculate the standard fluctuation range of each capability indicator based on historical training data, dynamically generate the threshold range of each indicator, and store it in a threshold management table or cache; Acquire the capability indicator data of the capability indicator set of the resident physician in real time and compare it with the threshold range; determine whether the capability indicator data exceeds the threshold range; if so, mark the abnormality and record the abnormality cause and indicator information; The analysis unit is used to take the ability indicator set of the current doctor, the marked ability label and the abnormal identification as input, and pass them into the LLM node of the workflow. Through the configured Prompt template, the large language model is called to generate the physician ability portrait and the report analysis of the regulated physician's ability performance analysis, standard comparison analysis, strengths and weaknesses analysis and improvement suggestions.

Images