Human anatomical structure recognition and scoring system and method based on multi-modal fusion
The multimodal fusion-based human anatomical structure recognition and scoring system solves the problems of multimodal input and image quality diagnosis, achieves fairness and transparency in scoring, promotes students' self-directed learning and group collaboration, and improves teaching effectiveness.
Patent Information
- Application Number
- CN202511578857.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-10-31
- Publication Date
- 2026-01-27
Smart Images
Figure CN121414548A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to image recognition and intelligent scoring, specifically to a system and method for human anatomical structure recognition and scoring based on multimodal fusion. Background Technology
[0002] With the accelerated digitalization and intelligentization of experimental teaching in higher medical colleges, the assessment methods for courses such as anatomy are gradually shifting from manual grading to automated scoring. Traditional anatomy exams or experimental assessments typically require students to identify, label, or explain anatomical structures, with teachers checking answers and assigning grades. This model has the following problems: 1) Manual grading is time-consuming and labor-intensive, making it unsuitable for large-scale teaching assessments; 2) Teachers' subjective factors have a significant impact, resulting in a lack of consistency and traceability in grading; 3) The difficulty of different test questions varies, leading to a lack of comparability in scores; 4) There is a lack of intelligent feedback and personalized learning support, making it difficult for students to receive timely and targeted improvement suggestions.
[0003] In recent years, with the development of deep learning and computer vision technologies, some progress has been made in the field of medical image recognition and intelligent scoring. For example, some studies have used convolutional neural networks (CNNs) to automatically identify organs or tissue structures in medical images, while others have used virtual simulation platforms to achieve visualized teaching of anatomy. However, these solutions are usually only designed for single-modal inputs (such as CT scans or single images), making it difficult to accommodate different imaging sources. Furthermore, they are mostly focused on the recognition itself and lack fair control mechanisms for scoring and score conversion in teaching assessments.
[0004] Furthermore, existing automatic scoring systems mostly output correct / incorrect judgments directly based on image recognition results, without considering the impact of image quality on recognition accuracy, and lack mechanisms for quality diagnosis and correction of student-submitted answer images. More importantly, current schemes lack "score equivalence processing," meaning they cannot standardize and convert scores according to different test papers or question difficulties, resulting in a lack of comparability between scores from different exam sessions, seriously affecting the fairness of grading and the accuracy of teaching assessment.
[0005] From a teaching perspective, existing automated grading systems often neglect student learning interaction and group collaboration, failing to establish a collaborative learning mechanism based on class profiles. Modern educational philosophies emphasize the combination of individual and group learning. Integrating AI grading systems with student profiles and LLM (Large Language Model) interpretation mechanisms can not only improve the objectivity of grading but also support personalized training and collaborative group learning, further promoting teaching interaction and enhancing learning outcomes. Summary of the Invention
[0006] (a) Technical problems to be solved In view of the above-mentioned shortcomings of the existing technology, the present invention provides a human anatomical structure recognition and scoring system and method based on multimodal fusion, which can effectively overcome the shortcomings of the existing technology, such as the inability to process multimodal input, lack of quality diagnosis and correction of answer images, lack of comparability of scores, and neglect of students' learning interaction and group collaboration.
[0007] (II) Technical Solution To achieve the above objectives, the present invention provides the following technical solution: A multimodal fusion-based human anatomical structure recognition and scoring system, comprising a server and a client, wherein the server includes: The image acquisition and processing module acquires anatomical structure images submitted by students for the exam papers to be graded. For multimodal inputs, it performs spatial registration and feature fusion to uniformly identify anatomical structures and obtain anatomical structure images. The image quality diagnosis module detects the clarity, framing, and occlusion of anatomical structure images, and prompts for reshooting or manual evaluation for anatomical structure images with substandard image quality. The anatomical structure recognition module inputs qualified anatomical structure images into the anatomical structure recognition model, extracts anatomical structure features, and identifies the location and category of the target structure. The annotation and explanation module uses a large language model (LLM) to interpret and analyze the model recognition results, determine the degree of matching with the standard answer, and generate semantic explanation text. The scoring module calculates the raw score for each question based on the degree of matching. The equivalence processing module standardizes and converts the raw scores of each question according to the difficulty of different test papers, thus achieving score equivalence processing under different test paper difficulties. The grade output and archiving module calculates and archives standardized grades, and uploads the scoring results to the client for subsequent sampling review and feedback.
[0008] Preferably, the image acquisition and processing module acquires anatomical structure images submitted by students for grading of exam papers. For multimodal input, it performs spatial registration and feature fusion to uniformly identify anatomical structures, obtaining anatomical structure images, including: For multimodal inputs including medical images, real specimen images, and standard digital anatomical atlases, spatial registration functions are used. Spatial registration is performed on each modal input, where T(x,y) is the new coordinate obtained after spatial registration, (x,y) is the coordinate in the original coordinate space, A and B are linear transformation coefficients, and C is the translation parameter; Utilizing multimodal feature weighting mechanism Feature fusion is performed on the features of each modality, where W i Let i be the weight of the i-th feature. f represents the i-th eigenvector. i With mean vector The square of the Euclidean distance between them The variance of the eigenvectors; Anatomical structures are identified in a unified coordinate space by performing spatial registration and feature fusion, resulting in anatomical structure images.
[0009] Preferably, the image quality diagnosis module detects the sharpness, framing, and occlusion of anatomical structure images, and prompts for reshooting or manual evaluation for anatomical structure images with substandard image quality, including: Clarity, an index of sharpness in anatomical structure images, is calculated using gradient variance. ; in, The gradient of the anatomical structure image. This is the variance calculation function; Simultaneously, the proportion of occluded areas in the anatomical structure image is detected. If either the clarity index or the proportion of occluded areas in the anatomical structure image exceeds the corresponding preset threshold, the image quality of the anatomical structure image is deemed unqualified, and a retake or manual evaluation is prompted.
[0010] Preferably, the anatomical structure recognition module inputs a qualified anatomical structure image into the anatomical structure recognition model, extracts anatomical structure features, and identifies the location and category of the target structure, including: An anatomical structure image with acceptable image quality is input into the YOLO and ViT collaborative anatomical structure recognition model. Spatial and texture information is extracted through multi-layer Transformer encoding and cross-modal attention mechanism, and the target structure's bounding box B_p, class label y_c, and confidence P_c of the bounding box B_p are output.
[0011] Preferably, the annotation and explanation module uses a Large Language Model (LLM) to interpret and analyze the model recognition results, determine the degree of matching with the standard answer, and generate semantic explanation text, including: The recognition overlap (IoU) between the recognition bounding box B_p and the standard bounding box B_t is calculated using the following formula to determine the degree of matching between the model's recognition result and the standard answer: ; in, This represents the area of the intersection between the recognition bounding box B_p and the standard bounding box B_t. This represents the area of the union between the recognition box B_p and the standard box B_t; Simultaneously, the Large Language Model (LLM) is used to generate natural language descriptions for semantic interpretation, producing semantic explanation texts to help teachers verify the reasons for students' misunderstandings. The semantic explanation texts are generated by the Large Language Model (LLM) to produce an explanation credibility index (ECI). ; Where P_c is the confidence score of the bounding box B_p, and Q is the quality score of the anatomical structure image. , , All are empirical weights, and ; When the Explanation Credibility Index (ECI) > 0.85, the semantic explanation text is considered credible; when the ECI < 0.6, it is automatically suggested that the teacher review it.
[0012] Preferably, the scoring module calculates the raw score for each question based on the degree of matching, including: The original score x of the question is calculated based on the identification overlap (IoU) matching: .
[0013] Preferably, the equivalence processing module standardizes the raw scores of each question according to different test paper difficulties, realizing score equivalence processing under different test paper difficulties, including: Using linear transformation functions Or a nonlinear transformation function applicable to non-normally distributed samples The raw scores for each question are standardized and converted to achieve score equivalence under different test paper difficulties; Standardized performance distribution forms a comparable curve in the class profile, which is used for subsequent personalized training recommendations and weight allocation of group collaborative learning. Where f(x) is the converted standard score for the question. , These are the mean and standard deviation of the original score distribution, respectively. , These are the mean and standard deviation of the transformed target distribution, respectively.
[0014] Preferably, the score output and archiving module statistically analyzes and archives standardized scores, and simultaneously uploads the scoring results to the client for subsequent sampling review and feedback, including: When conducting sampling review, a portion of the questions are automatically selected for manual review by teachers. If the review results are consistent, the review is marked as passed; otherwise, the teacher adjusts the standard score of the questions based on the deviation. After the teacher corrects the standard scores of the questions, the weights are optimized using gradient descent based on the corrected samples to form a closed loop for continuous improvement. ; in, The weight increment for the i-th feature is... The weight W of the scoring loss function L for the i-th feature is represented by this value. i gradient, This is the learning rate.
[0015] Preferably, the server and client exchange data through an HTTPS encrypted communication interface and use JSON structured message format to ensure information security and transmission efficiency; The client includes a student client, a teacher client, and a group collaborative learning module; The student-side interface is used for image acquisition, answering, and feedback. It dynamically pushes training questions based on student profiles through a personalized training recommendation interface, enabling adaptive question recommendation and training on areas of knowledge weakness. Student profile updates are represented by the following formula: ; Here, Profile_new represents the updated student profile, Profile_old represents the original student profile, and Result_vector represents the latest assessment results. Forgetting factor; The teacher's side is used for result verification and teaching statistics, and communicates with the student's side through an encrypted interface; The collaborative learning module dynamically creates learning groups based on class profiles, enabling tasks to be assigned and distributed, with group members answering and annotating questions, peer review and discussion, and teacher review and feedback, thus promoting a closed loop of collaborative learning and teaching.
[0016] A multimodal fusion-based method for human anatomical structure recognition and scoring includes the following steps: The system acquires anatomical structure images submitted by students for their exam papers, sourced from medical images, real specimen images, and standard digital anatomical atlases. For multimodal inputs, it performs spatial registration and feature fusion to uniformly identify anatomical structures and obtain anatomical structure images. The system detects the clarity, framing, and occlusion of anatomical structure images, and prompts for reshooting or manual evaluation for anatomical structure images that do not meet the image quality standards. Input high-quality anatomical structure images into the anatomical structure recognition model to extract anatomical structure features and identify the location and category of the target structure. The large language model (LLM) is used to interpret and analyze the model recognition results, determine the degree of matching with the standard answer, and generate semantic explanation text. Calculate the raw score for each question based on the degree of matching. The raw scores of each question are standardized and converted according to the different difficulty levels of the test paper to achieve equivalent score processing under different test paper difficulties; Standardized scores are compiled and archived, and the scoring results are uploaded to the client for subsequent sampling review and feedback.
[0017] (III) Beneficial Effects Compared with existing technologies, the human anatomical structure recognition and scoring system and method based on multimodal fusion provided by this invention integrates multi-source anatomical information and introduces image quality diagnosis, significantly improving image recognition accuracy and reducing misjudgments. It combines large language model (LLM) interpretation to construct a transparent and traceable scoring mechanism, while employing score equivalence processing and standard score mapping to ensure fairness across test papers. Through personalized training recommendation and group collaborative learning mechanisms, it promotes self-directed learning and enhances teaching interaction, ultimately forming a comprehensive teaching system that is fully automated, multimodal, interpretable in scoring, fair in results, and has a three-party feedback loop. Furthermore, it relies on HTTPS and data encryption to ensure security and compliance. Attached Figure Description
[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.
[0019] Figure 1 This is a schematic diagram of the system of the present invention; Figure 2 This is a schematic diagram illustrating the workflow of the image acquisition and processing module, the image quality diagnosis module, and the anatomical structure recognition module in this invention. Figure 3 This is a schematic diagram illustrating the workflow of the annotation and interpretation module, the scoring module, and the equivalence processing module in this invention. Figure 4 This is a schematic diagram of the fraction equivalence processing module of the present invention performing fraction equivalence processing. Figure 5 This is a schematic diagram of the sampling verification process in this invention; Figure 6 This is a schematic diagram of the personalized training process in this invention; Figure 7 This is a schematic diagram of group collaborative learning in this invention. Detailed Implementation
[0020] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.
[0021] The following describes the specific architecture of the human anatomical structure recognition and scoring system based on multimodal fusion provided by this invention, using specific examples (e.g.) Figure 1 (As shown): It includes a server and a client, wherein the server includes the following functional modules: The image acquisition and processing module acquires anatomical structure images submitted by students for the exam papers to be graded. For multimodal inputs, it performs spatial registration and feature fusion to uniformly identify anatomical structures and obtain anatomical structure images. The image quality diagnosis module detects the clarity, framing, and occlusion of anatomical structure images, and prompts for reshooting or manual evaluation for anatomical structure images with substandard image quality. The anatomical structure recognition module inputs qualified anatomical structure images into the anatomical structure recognition model, extracts anatomical structure features, and identifies the location and category of the target structure. The annotation and explanation module uses a large language model (LLM) to interpret and analyze the model recognition results, determine the degree of matching with the standard answer, and generate semantic explanation text. The scoring module calculates the raw score for each question based on the degree of matching. The equivalence processing module standardizes and converts the raw scores of each question according to the difficulty of different test papers, thus achieving score equivalence processing under different test paper difficulties. The grade output and archiving module calculates and archives standardized grades, and uploads the scoring results to the client for subsequent sampling review and feedback.
[0022] I. Image Acquisition and Processing Module like Figure 2 As shown, the image acquisition and processing module acquires anatomical structure images submitted by students for grading. For multimodal input, it performs spatial registration and feature fusion to uniformly identify anatomical structures, resulting in anatomical structure images, including: For multimodal inputs including medical images, real specimen images, and standard digital anatomical atlases, spatial registration functions are used. Spatial registration is performed on each modal input, where T(x,y) is the new coordinate obtained after spatial registration, (x,y) is the coordinate in the original coordinate space, A and B are linear transformation coefficients, and C is the translation parameter; Utilizing multimodal feature weighting mechanism Feature fusion is performed on the features of each modality, where W i Let i be the weight of the i-th feature. f represents the i-th eigenvector. i With mean vector The square of the Euclidean distance between them The variance of the eigenvectors; Anatomical structures are identified in a unified coordinate space by performing spatial registration and feature fusion, resulting in anatomical structure images.
[0023] II. Image Quality Diagnosis Module like Figure 2 As shown, the image quality diagnosis module detects the sharpness, framing, and occlusion of anatomical structure images. For anatomical structure images with substandard image quality, it prompts for reshooting or manual evaluation, including: Clarity, an index of sharpness in anatomical structure images, is calculated using gradient variance. ; in, The gradient of the anatomical structure image. This is the variance calculation function; Simultaneously, the proportion of occluded areas in the anatomical structure image is detected. If either the clarity index or the proportion of occluded areas in the anatomical structure image exceeds the corresponding preset threshold, the image quality of the anatomical structure image is deemed unqualified, and a retake or manual evaluation is prompted.
[0024] III. Anatomical Structure Recognition Module like Figure 2 As shown, the anatomical structure recognition module inputs high-quality anatomical structure images into the anatomical structure recognition model, extracts anatomical structure features, and identifies the location and category of the target structure, including: An anatomical structure image with acceptable image quality is input into an anatomical structure recognition model that combines YOLO and ViT (Vision Transformer). Spatial and texture information is extracted through multi-layer Transformer encoding and cross-modal attention mechanism, and the target structure's bounding box B_p, class label y_c, and confidence score P_c of the bounding box B_p are output.
[0025] IV. Annotation and Explanation Module like Figure 3 As shown, the annotation and explanation module uses a Large Language Model (LLM) to interpret and analyze the model's recognition results, determine the degree of matching with the standard answer, and generate semantic explanation text, including: The recognition overlap (IoU) between the recognition bounding box B_p and the standard bounding box B_t is calculated using the following formula to determine the degree of matching between the model's recognition result and the standard answer: ; in, This represents the area of the intersection between the recognition bounding box B_p and the standard bounding box B_t. This represents the area of the union between the recognition box B_p and the standard box B_t; Simultaneously, the Large Language Model (LLM) is used to generate natural language descriptions for semantic interpretation, producing semantic explanation texts to help teachers verify the reasons for students' misunderstandings. The semantic explanation texts are generated by the Large Language Model (LLM) to produce an explanation credibility index (ECI). ; Where P_c is the confidence score of the bounding box B_p, and Q is the quality score of the anatomical structure image. , , All are empirical weights, and ; When the Explanation Credibility Index (ECI) > 0.85, the semantic explanation text is considered credible; when the ECI < 0.6, it is automatically suggested that the teacher review it.
[0026] V. Scoring Module like Figure 3 As shown, the scoring module calculates the raw score for each question based on the degree of matching, including: The original score x of the question is calculated based on the identification overlap (IoU) matching: .
[0027] VI. Equivalence Processing Module like Figure 3 and Figure 4 As shown, the equivalence processing module standardizes the raw scores of each question according to different test paper difficulties, realizing score equivalence processing under different test paper difficulties, including: Using linear transformation functions Or a nonlinear transformation function applicable to non-normally distributed samples The raw scores for each question are standardized and converted to achieve score equivalence under different test paper difficulties; Standardized performance distribution forms a comparable curve in the class profile, which is used for subsequent personalized training recommendations and weight allocation of group collaborative learning. Where f(x) is the converted standard score for the question. , These are the mean and standard deviation of the original score distribution, respectively. , These are the mean and standard deviation of the transformed target distribution, respectively.
[0028] VII. Score Output and Archiving Module The score output and archiving module calculates and archives standardized scores, and simultaneously uploads the scoring results to the client for subsequent sampling review and feedback, including: like Figure 5 As shown, during the sampling review, a portion of the questions are automatically selected for manual review by the teachers. If the review results are consistent, the review is marked as passed; otherwise, the teachers adjust the standard scores of the questions based on the deviations. After the teacher corrects the standard scores of the questions, the weights are optimized using gradient descent based on the corrected samples to form a closed loop for continuous improvement. ; in, The weight increment for the i-th feature is... The weight W of the scoring loss function L for the i-th feature is represented by this value. i gradient, This is the learning rate.
[0029] In the technical solution of this application, the server and the client exchange data through an HTTPS encrypted communication interface and adopt JSON structured message format to ensure information security and transmission efficiency; The client includes a student client, a teacher client, and a group collaborative learning module; The student-side interface is used for image acquisition, answering, and feedback. It dynamically pushes training questions based on student profiles through a personalized training recommendation interface, enabling adaptive question recommendation and training on areas of knowledge weakness. Student profile updates are represented by the following formula: ; Here, Profile_new represents the updated student profile, Profile_old represents the original student profile, and Result_vector represents the latest assessment results. Forgetting factor; The teacher's side is used for result verification and teaching statistics, and communicates with the student's side through an encrypted interface; The collaborative learning module dynamically creates learning groups based on class profiles, enabling tasks to be assigned and distributed, with group members answering and annotating questions, peer review and discussion, and teacher review and feedback, thus promoting a closed loop of collaborative learning and teaching.
[0030] Figure 6This is a flowchart illustrating the personalized training process in this invention, describing the student interaction flow and personalized learning mechanism. Students log in to the student client (A1), select a learning chapter (A2), and the system recommends questions based on the student's profile (A3). After the student completes anatomical annotations or answers (A4), the image is uploaded and triggers a background process (A5), at which point a multimodal fusion-based human anatomical structure recognition and scoring method is executed. The scoring result is returned to the student client (A6), and the system simultaneously displays the standard answer and anatomical explanation (A7) and updates the student's profile (A8).
[0031] Figure 7 This diagram illustrates collaborative learning in groups, achieving a collaborative scoring and teaching loop at the class level. The system automatically generates learning groups (G1) based on class profiles and complementarity, and assigns tasks (G2). Students within the group complete their answers and annotations (G3), images are uploaded, and a background process is triggered. At this point, a multimodal fusion-based human anatomical structure recognition and scoring method is executed. The system allows group members to peer review and discuss (G4). During this process, the annotation and explanation module can be called to generate semantic explanation text to aid understanding. The teacher reviews the work (G5), checking the group discussion results and peer review records, modifying the final score or adding comments if necessary, thus achieving a closed-loop management model of "AI automatic scoring + human-computer collaborative review."
[0032] Based on the aforementioned human anatomical structure recognition and scoring system based on multimodal fusion, this application also discloses a human anatomical structure recognition and scoring method based on multimodal fusion, comprising the following steps: The system acquires anatomical structure images submitted by students for their exam papers, sourced from medical images, real specimen images, and standard digital anatomical atlases. For multimodal inputs, it performs spatial registration and feature fusion to uniformly identify anatomical structures and obtain anatomical structure images. The system detects the clarity, framing, and occlusion of anatomical structure images, and prompts for reshooting or manual evaluation for anatomical structure images that do not meet the image quality standards. Input high-quality anatomical structure images into the anatomical structure recognition model to extract anatomical structure features and identify the location and category of the target structure. The large language model (LLM) is used to interpret and analyze the model recognition results, determine the degree of matching with the standard answer, and generate semantic explanation text. Calculate the raw score for each question based on the degree of matching. The raw scores of each question are standardized and converted according to the different difficulty levels of the test paper to achieve equivalent score processing under different test paper difficulties; Standardized scores are compiled and archived, and the scoring results are uploaded to the client for subsequent sampling review and feedback.
[0033] The technical solution proposed in this application is ready for application in medical experimental courses in universities. In actual experiments, the trimodal fusion recognition technology increases the scoring speed by about 90% and reduces the manual review rate to less than 5%. The score equivalence processing ensures that the comparability difference between different test paper scores is less than 2%. The group collaborative learning module significantly improves student participation, with the average number of peer reviews increasing by 1.8 times. The annotation and explanation module makes it more intuitive for students to locate errors and understand the reasons, and reduces the average grading time of teachers by more than 60%.
[0034] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions will not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A human anatomical structure recognition and scoring system based on multimodal fusion, characterized in that: Includes a server and a client, wherein the server includes: The image acquisition and processing module acquires anatomical structure images submitted by students for the exam papers to be graded. For multimodal inputs, it performs spatial registration and feature fusion to uniformly identify anatomical structures and obtain anatomical structure images. The image quality diagnosis module detects the clarity, framing, and occlusion of anatomical structure images, and prompts for reshooting or manual evaluation for anatomical structure images with substandard image quality. The anatomical structure recognition module inputs qualified anatomical structure images into the anatomical structure recognition model, extracts anatomical structure features, and identifies the location and category of the target structure. The annotation and explanation module uses a large language model (LLM) to interpret and analyze the model recognition results, determine the degree of matching with the standard answer, and generate semantic explanation text. The scoring module calculates the raw score for each question based on the degree of matching. The equivalence processing module standardizes and converts the raw scores of each question according to the difficulty of different test papers, thus achieving score equivalence processing under different test paper difficulties. The grade output and archiving module calculates and archives standardized grades, and uploads the scoring results to the client for subsequent sampling review and feedback.
2. The human anatomical structure recognition and scoring system based on multimodal fusion according to claim 1, characterized in that: The image acquisition and processing module acquires anatomical structure images submitted by students for grading. For multimodal input, it performs spatial registration and feature fusion to uniformly identify anatomical structures, resulting in anatomical structure images, including: For multimodal inputs including medical images, real specimen images, and standard digital anatomical atlases, spatial registration functions are used. Spatial registration is performed on each modal input, where T(x,y) is the new coordinate obtained after spatial registration, (x,y) is the coordinate in the original coordinate space, A and B are linear transformation coefficients, and C is the translation parameter; Utilizing multimodal feature weighting mechanism Feature fusion is performed on the features of each modality, where W i Let i be the weight of the i-th feature. f represents the i-th eigenvector. i With mean vector The square of the Euclidean distance between them The variance of the eigenvectors; Anatomical structures are identified in a unified coordinate space by performing spatial registration and feature fusion, resulting in anatomical structure images.
3. The human anatomical structure recognition and scoring system based on multimodal fusion according to claim 2, characterized in that: The image quality diagnosis module detects the sharpness, framing, and occlusion of anatomical structure images. For anatomical structure images with substandard image quality, it prompts for reshooting or manual evaluation, including: Clarity, an index of sharpness in anatomical structure images, is calculated using gradient variance. ； in, The gradient of the anatomical structure image. This is the variance calculation function; Simultaneously, the proportion of occluded areas in the anatomical structure image is detected. If either the clarity index or the proportion of occluded areas in the anatomical structure image exceeds the corresponding preset threshold, the image quality of the anatomical structure image is deemed unqualified, and a retake or manual evaluation is prompted.
4. The human anatomical structure recognition and scoring system based on multimodal fusion according to claim 3, characterized in that: The anatomical structure recognition module inputs high-quality anatomical structure images into the anatomical structure recognition model, extracts anatomical structure features, and identifies the location and category of the target structure, including: An anatomical structure image with acceptable image quality is input into the YOLO and ViT collaborative anatomical structure recognition model. Spatial and texture information is extracted through multi-layer Transformer encoding and cross-modal attention mechanism, and the target structure's bounding box B_p, class label y_c, and confidence P_c of the bounding box B_p are output.
5. The human anatomical structure recognition and scoring system based on multimodal fusion according to claim 4, characterized in that: The annotation and explanation module uses a Large Language Model (LLM) to interpret and analyze the model recognition results, determine the degree of matching with the standard answer, and generate semantic explanation text, including: The recognition overlap (IoU) between the recognition bounding box B_p and the standard bounding box B_t is calculated using the following formula to determine the degree of matching between the model's recognition result and the standard answer: ； in, This represents the area of the intersection between the recognition bounding box B_p and the standard bounding box B_t. This represents the area of the union between the recognition box B_p and the standard box B_t; Simultaneously, the Large Language Model (LLM) is used to generate natural language descriptions for semantic interpretation, producing semantic explanation texts to help teachers verify the reasons for students' misunderstandings. The semantic explanation texts are generated by the Large Language Model (LLM) to produce an explanation credibility index (ECI). ； Where P_c is the confidence score of the bounding box B_p, and Q is the quality score of the anatomical structure image. , , All are empirical weights, and ; When the Explanation Credibility Index (ECI) > 0.85, the semantic explanation text is considered credible; when the ECI < 0.6, it is automatically suggested that the teacher review it.
6. The human anatomical structure recognition and scoring system based on multimodal fusion according to claim 5, characterized in that: The scoring module calculates the raw score for each question based on the degree of matching, including: The original score x of the question is calculated based on the identification overlap (IoU) matching: 。 7. The human anatomical structure recognition and scoring system based on multimodal fusion according to claim 6, characterized in that: The equivalence processing module standardizes the raw scores of each question according to different test paper difficulties, realizing score equivalence processing under different test paper difficulties, including: Using linear transformation functions Or a nonlinear transformation function applicable to non-normally distributed samples The raw scores for each question are standardized and converted to achieve score equivalence under different test paper difficulties; Standardized performance distribution forms a comparable curve in the class profile, which is used for subsequent personalized training recommendations and weight allocation of group collaborative learning. Where f(x) is the converted standard score for the question. , These are the mean and standard deviation of the original score distribution, respectively. , These are the mean and standard deviation of the transformed target distribution, respectively.
8. The human anatomical structure recognition and scoring system based on multimodal fusion according to claim 7, characterized in that: The score output and archiving module statistically analyzes and archives standardized scores, and simultaneously uploads the scoring results to the client for subsequent sampling review and feedback, including: When conducting sampling review, a portion of the questions are automatically selected for manual review by teachers. If the review results are consistent, the review is marked as passed; otherwise, the teacher adjusts the standard score of the questions based on the deviation. After the teacher corrects the standard scores of the questions, the weights are optimized using gradient descent based on the corrected samples to form a closed loop for continuous improvement. ； in, The weight increment for the i-th feature is... The weight W of the scoring loss function L on the i-th feature is represented by the following expression. i gradient, This is the learning rate.
9. The human anatomical structure recognition and scoring system based on multimodal fusion according to claim 1, characterized in that: The server and client exchange data through an HTTPS encrypted communication interface and use JSON structured message format to ensure information security and transmission efficiency. The client includes a student client, a teacher client, and a group collaborative learning module; The student-side interface is used for image acquisition, answering, and feedback. It dynamically pushes training questions based on student profiles through a personalized training recommendation interface, enabling adaptive question recommendation and training on areas of knowledge weakness. Student profile updates are represented by the following formula: ； Here, Profile_new represents the updated student profile, Profile_old represents the original student profile, and Result_vector represents the latest assessment results. Forgetting factor; The teacher's side is used for result verification and teaching statistics, and communicates with the student's side through an encrypted interface; The collaborative learning module dynamically creates learning groups based on class profiles, enabling tasks to be assigned and distributed, with group members answering and annotating questions, peer review and discussion, and teacher review and feedback, thus promoting a closed loop of collaborative learning and teaching.
10. A method for recognizing and scoring human anatomical structures based on multimodal fusion, applied to the human anatomical structure recognition and scoring system based on multimodal fusion as described in claim 1, characterized in that: Includes the following steps: The system acquires anatomical structure images submitted by students for their exam papers, sourced from medical images, real specimen images, and standard digital anatomical atlases. For multimodal inputs, it performs spatial registration and feature fusion to uniformly identify anatomical structures and obtain anatomical structure images. The system detects the clarity, framing, and occlusion of anatomical structure images, and prompts for reshooting or manual evaluation for anatomical structure images that do not meet the image quality standards. Input high-quality anatomical structure images into the anatomical structure recognition model to extract anatomical structure features and identify the location and category of the target structure. The large language model (LLM) is used to interpret and analyze the model recognition results, determine the degree of matching with the standard answer, and generate semantic explanation text. Calculate the raw score for each question based on the degree of matching. The raw scores of each question are standardized and converted according to the different difficulty levels of the test paper to achieve equivalent score processing under different test paper difficulties; Standardized scores are compiled and archived, and the scoring results are uploaded to the client for subsequent sampling review and feedback.