Educational performance evaluation system and method indicating the achievement level of learning and program outcomes

Abstract

This invention relates to a system and method capable of indicating the level of achievement of students in program outcomes, evaluating the performance of instructors in a course under various categories such as course, question, and learning outcomes, and communicating with course information packages. Program outcomes comprise all the knowledge, skills, and behavioral components required to achieve the educational objectives of a program, including the outcomes specified by the relevant accreditation body. Program outcomes must be consistent with the educational objectives and uniquely defined for each programThe invention relates to a computer-based educational performance evaluation system and method that indicates the achievement level of learning and program outcomes.

Description

[0001] EDUCATIONAL PERFORMANCE EVALUATION SYSTEM AND METHOD INDICATING THE ACHIEVEMENT LEVEL OF LEARNING AND PROGRAM OUTCOMES Technical Field of the Invention:

[0002] The invention relates to a computer-based educational performance evaluation system and method that indicates the achievement level of learning and program outcomes. The invention particularly relates to an educational performance evaluation system and method that indicates the achievement level of students in program outcomes, determines the performance of instructors in a course under various headings such as course, question, and learning outcomes, and communicates with course information packages.

[0003] State of the Art:: Program outcomes comprise all the knowledge, skills, and behavioral components required to achieve the educational objectives of a program, including the outcomes specified by the relevant accreditation body. Program outcomes must align with educational objectives and be uniquely defined for each program.

[0004] Accreditation bodies require the measurement of the achievement level of program outcomes, as defined by departments or programs in universities, on a student-by- student basis. In this context, for example, the Engineering Education Programs Evaluation and Accreditation Association (MUDEK) specifies in its evaluation criteria handbook under criterion 3.2: "A measurement and evaluation process must be established and operated to determine and document the level of achievement of program outcomes periodically." Additionally, criterion 3.3 states: "Engineering programs must demonstrate that their students nearing graduation have achieved the program outcomes." To meet these criteria, universities require software integrated with their student grading systems. Additionally, there is no comprehensive system in higher education institutions to measure the overall success of the curricula they have developed. In the guidelines of organizations authorized for institutional and program accreditation, the program design, evaluation, and updating section thoroughly examines how the process is monitored and evaluated at an institutional level. Higher Education Quality Council (YOKAK) is monitored with six sub-criteria in criterion B.1 under the title of Education and Training in the Institutional Internal Evaluation Report (IER) Preparation Guide.

[0005] There are existing educational performance evaluation systems and methods in the known state of the art. However, these systems and methods have certain technical shortcomings. Below are some inventions for which patent applications have been filed in the known state of the art.

[0006] The invention subject to the application numbered "TR202022546A2" in the known state of the art is an Ahi competency-based education portal capable of performing item analysis of exam questions. The system comprises an assessment and evaluation module that analyzes the answers entered by students into the database through the processor after an exam is conducted via the system. This module performs item analysis of the questions (e.g., difficulty, discrimination) and allows administrators to deactivate unsuitable questions, preventing their reuse.

[0007] The requirements specified in the application numbered TR202022546A2 correspond to the electronic information package system used by many universities. This system includes details such as the learning outcomes of each course, program outcomes, course contents, topics, related course grades, and information about the instructor teaching the course. The application numbered TR202022546A2 has developed a system that performs item analysis on exam questions in addition to this electronic information package. Item analysis determines the difficulty level and discriminatory characteristics of the exam questions. This process is entirely different from the process of our invention. In the method of the invention detailed in the section below, item analysis of questions is not performed.

[0008] Additionally, the system is only suitable for online education and evaluations using multiple-choice questions. However, education and training activities in higher education are conducted face-to-face. In addition, the fact that practical training is active limits the application of this software. Additionally, the exam system in the patent numbered TR202022546A2 is based on online exams. The patent application numbered TR202022546A2, included in the current state of the art, is an electronic information package system used by many universities. This system includes information on each course's learning outcomes, program outcomes, course contents, topics, relevant course grades, and details about the course's instructor. The application numbered TR202022546A2 has developed a system that performs item analysis on exam questions in addition to this system. Item analysis determines the difficulty level and discriminative properties of exam questions. However, in the system and method described in detail in the detailed explanations of the invention, item analysis of questions is not performed. The system and method of the invention, in addition to performing all processes except item analysis included in the application numbered TR202022546A2, also calculates the level of attainment of learning outcomes (LOs) and program outcomes (POs) in the case of face-to-face education, open-ended exams, and practical training. This situation differs from the display of LOs and POs on an information system. As mentioned earlier, LO and PO matrices and their relationships exist in all universities. In the system and method of the invention, the level of achievement of LOs is calculated by matching the scores students received from exam questions with the LOs assigned to those questions by the instructor. Subsequently, the system calculates the level of achievement of POs using the LO x PO matrices available in the information package of each university. The system can perform this process separately for each student, each course, and each instructor. Additionally, once the evaluation for each course is completed, a departmental report card is generated. Similarly, it provides the ability to monitor the status of faculties and the entire institution. Unlike the TR202022546A2 application, the system and method of the invention do not perform any calculations regarding the discriminative or difficulty levels of the questions, as done in the item analysis of the mentioned application. Item analysis and determining the achievement levels of LOs and POs are entirely different concepts. Additionally, in the TR202022546A2 application, learning outcomes can only be obtained for online courses. In contrast, the system and method of the invention can determine the impact of in-person exams on learning outcomes, thereby enabling the evaluation of program outcome performance. In in-person exams, students' question-based scores are recorded on optical forms placed on paper, and these forms are transferred to a virtual environment using optical scanners. Thanks to the differences between the invention and the TR202022546A2 application, the levels at which students achieve learning outcomes (LOs) and program outcomes (POs) can be determined. To reiterate, LOs and POs are data available in the electronic information systems of all universities. These are accessible to everyone. However, the extent to which students achieve the declared learning outcomes (LOs) and program outcomes (POs) can be determined using the system and method of the invention. This feature is not present in the TR202022546A2 application.

[0009] The invention described in the "US20060147890A1" application in the prior art discusses a system and method for effectively measuring and reporting performance based on learning outcomes defined for students, professors, classes, courses, and programs. A learning content item associated with a learning outcome indicator, an assessment content item to evaluate student performance according to the learning outcome, and an indicator displaying student performance based on the learning outcome are presented on the student terminal. In the application numbered US20060147890A1 , the item analysis cannot demonstrate the level of students' attainment of LOs and POs. Item analysis is a method used to determine whether the questions asked in exams are appropriate for the course. The system and method described in the invention, with details provided in the detailed description section, differ from the application numbered US20060147890A1 in that it not only allows the attainment level of program outcomes to be determined on a course basis but also enables the attainment level of program outcomes to be viewed on a unit and faculty basis. Additionally, with this method, graduate competencies are also monitored. In the patent application numbered US20060147890A1 , performance is measured based on learning outcomes, and academic staff performance is also evaluated. However, unlike the system and method described in the invention, as detailed in the detailed description section, program outcomes are associated with learning outcomes, allowing attainment to be determined not only for individual courses but also at the unit level, in light of all results. This result not only provides performance evaluation but also enables the measurement of graduate competencies at both the unit and faculty levels. Graduate competency, which is the ability to meet program outcomes, is a critical aspect in terms of accreditation qualifications. Answers to such parameters are provided through this online program design. Such parameters are not addressed in the US20060147890A1 patent. While the US20060147890A1 patent includes learning outcomes for each course, the level of attainment for program outcomes and their overall objectives are generally not included in the patents in the prior art. Additionally, in the US20060147890A1 application, learning outcomes can only be obtained for online courses. A course-program outcomes evaluation survey is conducted with students at the end of each semester for courses offered during that term. In this system, students attempt to measure the level of attainment of program outcomes by establishing a connection between program outcomes and courses. However, since survey results do not rely on quantitative data like student grades, accreditation organizations do not accept them. In the system and method described in the invention, the impact of face- to-face exams on learning outcomes can also be determined, enabling the calculation of program outcome performance. In face-to-face exams, students' question-based scores are recorded on optical forms placed on paper, and these forms are transferred to a digital environment using optical readers.

[0010] The invention described in the prior art under application number "US2018225981" defines a system for managing learning program outcomes. This system ensures the effective management of program outcomes and the assessment of student performance through the use of various components. The invention addresses this by employing a system that stores a student's overall performance for a learning objective selected at the program, flow, or course level in an electronic database. In application number "US2018225981 ," no information is provided regarding communication between different modules. It is merely an electronic database system that records data such as student grades and exams. However, the differences in the system and method of the invention, as detailed in the Detailed Description section, include the capability to determine the level of attainment of program outcomes on a course basis, as well as the ability to view the attainment of program outcomes on a unit and faculty basis. Additionally, this method enables continuous communication with different modules such as the graduate competencies module, survey system, ECTS calculation system, and grade entry system. In this way, learning and program outcomes can be presented to the user in an integrated manner. Additionally, with this method, the performance score of the instructor can be calculated using various parameters such as graduate competencies, the survey system, the ECTS calculation system, and the grade entry system. The system described in US2018225981 is merely an electronic database system that records data such as student grades and exams. There is no correlation or interaction specified between the recorded systems. However, unlike this, the system and method of the invention, as detailed in the detailed description section, associate learning outcomes with program outcomes, allowing access to both course-specific and overall results at the unit level. This result not only provides performance evaluation but also enables the measurement of graduate competencies at both the unit and faculty levels. Graduate competency, which is the ability to achieve program outcomes, is a critical aspect of accreditation standards. Such parameters are addressed through this online program design. However, the system described in US2018225981 does not offer a solution for such parameters. The differences between the invention's system and method, detailed in the detailed description section, and the system described in US2018225981 allow for determining the extent to which students achieve learning outcomes (LOs) and program outcomes (POs). To reiterate, LOs and POs are data available in every university's electronic information systems and accessible to everyone. However, the degree to which students achieve the declared learning and program outcomes can be determined through the invention's system and method, as described in the detailed description. This feature is absent in the system described in US2018225981 .

[0011] The invention described in the application numbered 'WO2012002799," found in the known state of the art, relates to a system for managing outcome-based education for higher education academic programs. The system includes at least one design process, implementation process, and evaluation process, encompassing a planning structure to achieve the required learning outcomes. In the application numbered WO201 2002799, the grades students receive in exams are entered into the system to observe the level of attainment of educational outcomes. This system resembles the standard grade entry system of every university. However, the most significant difference between the system and method described in the invention, as detailed in the detailed description section, and the application numbered WO2012002799 is that, instead of entering students' cumulative grades from exams, the grades for each individual exam question are entered into the system. Since each exam question is matched with specific learning outcomes, the learning outcomes produce a distinct result from the students' overall exam grades. In contrast, in the application numbered WO201 2002799, exam results directly represent the outcomes of the learning objectives. Since this approach produces the same result as the student's exam grade, it does not provide an accurate evaluation. Due to the differences between the two systems, the level of attainment of students' learning and program outcomes can be measured as a distinct value, separate from their exam grades. This value provides a more comprehensive and detailed evaluation outcome. Additionally, with the ECTS measurement module integrated into the system, the students' efforts during the process of attaining program outcomes are assessed, and the ECTS for the course is calculated accordingly. This allows for a comparison between the ECTS credits defined in the curriculum and the actual ECTS achieved by the student.

[0012] Another technique used in the state of the art is the graduate survey. In this survey, students who have graduated from the department evaluate the relationship between program outcomes and courses from a graduate's perspective. However, as this technique relies on a survey and the results may vary depending on different variables, it is not accepted by accreditation bodies.

[0013] In some techniques, the scores students receive in courses are directly matched with program outcomes to determine the level of achievement of program outcomes. However, in this case, learning outcomes and their impacts on program outcomes are overlooked. On the other hand, the aforementioned techniques are solely aimed at measuring the level of achievement of program outcomes. Through the invention, the level of achievement of each course's learning outcomes (LOs) and program outcomes (POs) can be determined on a student-by-student basis. Additionally, the course introduction files of the instructor are uploaded to the system, the LO-PO relationships are retrieved through the Bologna system, and matched with the course's question-based scores. This enables an assessment of the instructor's overall performance. Moreover, the instructor can submit suggestions for continuous improvement through the software. A single software system has been developed that generally displays the performance of the instructor and the level of achievement of program outcomes by the student. This software provides all the evidence required by accreditation organizations regarding the level of attainment of program outcomes. The methods used in the prior art do not mention functionality such as deriving the relationship between learning outcomes (LOs) and program outcomes (POs) through the Bologna system and calculating the level of achievement of program outcomes based on this relationship. Although some inventions claim compatibility with the existing systems of educational institutions, there is no mention of communication features with various systems used by universities, such as the course grading system, Bologna information package, and quality management system.

[0014] In the prior art, there are methods that provide information on calculations. However, no software has been identified that establishes a relationship between students' question-based scores, learning outcomes (LOs), and program outcomes (POs), and displays the level of achievement of program outcomes. In some universities, program outcomes are determined at the departmental level. However, no software has been found that covers all courses across the university and establishes a relationship between the course information package of the instructor and the program outcomes. Previous techniques determine the level of achievement of program outcomes using various calculation methods.

[0015] In conclusion, due to the aforementioned shortcomings and the inadequacy of existing solutions on the subject, it has become necessary to develop an improvement in the relevant technical field.

[0016] Objective of the Invention::

[0017] The primary objective of the invention is to develop a system and method capable of evaluating educational performance by displaying the level of students' achievement of program outcomes, assessing the teaching performance of instructors under various categories such as course, question, and learning outcomes, and integrating with and communicating through course information packages.

[0018] The invention includes a method that enables the following objectives to be achieved:

[0019] Entry of course scores into the system on a question-by-question basis, Determining the level of achievement of learning outcomes (LOs) by entering the contribution of each question to LOs into the system, • Calculating the level of achievement of program outcomes (POs) by retrieving the LO- PO relationships through the Bologna system,

[0020] • Displaying the level of achievement of LOs and POs on a student basis,

[0021] • Determining the level of achievement of LOs on a course basis,

[0022] • Determining the level of achievement of POs on a program or department basis,

[0023] • Identifying the extent to which each course impacts the POs,

[0024] • Communicating with different systems used by the university, such as the grading system, Bologna information package, and quality management system,

[0025] • Allowing instructors to upload their course description files onto the system,

[0026] • Enabling instructors to submit continuous improvement suggestions through the system,

[0027] • Reporting the performance scores of all instructors within a department,

[0028] • Assigning an educational committee through the system to review and score the performed activities,

[0029] • Determining the performance score of an instructor by evaluating LO achievement levels, course description files, continuous improvement suggestions, and LO-PO matrix alignment,

[0030] • Automatically transferring grades entered into the grading system to the performance evaluation system,

[0031] • Redirecting continuous improvement suggestions as Corrective and Preventive Actions (CPAs) to the quality management system,

[0032] • Displaying the overall performance status of all programs on the main dashboard,

[0033] • Automatically pulling the results of student surveys into the system and reflecting survey scores on the instructor's performance,

[0034] • T ransferring the results of ECTS workload surveys into the system to compare the calculated ECTS with the actual ECTS of the course.

[0035] Another objective of the invention is to focus specifically on evaluating students' level of achievement of program outcomes (POs) and the performance of instructors. This focus provides more direct access to the data and analyses required in accreditation processes.

[0036] Another objective of the invention is to analyze students' achievement levels in detail by examining their performance on a question-by-question basis to determine their attainment of learning outcomes (LOs) and, consequently, program outcomes (POs). This provides richer data for identifying students' strengths and weaknesses and for evaluating the effectiveness of courses and programs.

[0037] Another objective of the invention is to automate data collection and evaluation processes by integrating with university systems. This approach saves time and enhances data accuracy. Another objective of the invention is to enable instructors to match exam questions with learning outcomes, analyze data received from optical readers, and transfer it to the database. Additionally, the ability to determine each question's contribution to students' learning outcomes allows for a more detailed assessment of the achievement levels of learning outcomes, providing enhanced functionality in this regard.

[0038] Another objective of the invention is to enable online access to learning and program outcomes, facilitate the evaluation of educational programs' compliance with the standards of the European Higher Education Area, and provide an advantage in terms of alignment with international standards. Another objective of the invention is to enhance integration capabilities, streamline data flow and processing procedures, and provide a more compatible working environment with the existing infrastructures of educational institutions.

[0039] Description of Figures:

[0040] FIGURE 1 : A drawing that provides an overview of the general workflow of the system and method described in the invention.

[0041] FIGURE 2: A drawing that illustrates the workflow of the Exam-LO success cycle of the system and method described in the invention. FIGURE 3: A drawing that illustrates the workflow of course PO success within the system and method described in the invention.

[0042] FIGURE 4: A drawing that illustrates the workflow of the PO success calculation cycle within the system and method described in the invention.

[0043] Reference Numbers:

[0044] 110. Course Introduction Module

[0045] 111. Course Introduction Database

[0046] 120. Assessment and Evaluation Module

[0047] 121. Assessment and Evaluation Database

[0048] 130. Education Commission System

[0049] 140. Instructor Performance System

[0050] 150. Student Survey Module

[0051] 160. Quality Management System

[0052] 1000. Entry of Exam Scores

[0053] 1001 . Determination of Question Success Status

[0054] 1002. Identification of Exam LO Success

[0055] 1003. Calculation of Question x LO Relationship

[0056] 1004. Retrieval of LO*PO Matrix from the Bologna Information Package

[0057] 1005. Entry of Course PO Success

[0058] 1006. Entry of Course LO Success

[0059] 1007. Exam 1 LO Success 1008. Exam 2 LO Success

[0060] 1009. Exam (n) LO Success

[0061] 1010. Retrieval of Courses' ECTS Workload from the Bologna Information Package

[0062] 101 1. Entry of Program's Course PO Success 1012. Course 1 PO Success

[0063] 1013. Course 2 PO Success

[0064] 1014. Course (n) PO Success

[0065] Description of the Invention: The invention relates to a computer-based education and training performance evaluation system and method that demonstrates the level of achievement of learning and program outcomes. In this invention, calculations are performed by a computation module located on a computer (server). Data is retrieved from the course introduction database (1 11 ) and the assessment and evaluation database (121 ), and the results and outputs are provided to the user through an interface. Users can view the results via the Internet and perform calculations on a remote computer (server).

[0066] The invention is an online portal that calculates the level of achievement of learning outcomes on a course and student basis, as well as the level of achievement of program outcomes. Additionally, it is an online system that allows the instructor to create course information packages and develop continuous improvement plans.

[0067] In the invention, exams are entered into the system on a course basis according to a specific matrix and are associated with learning outcomes. Subsequently, the learning outcomes are linked to program outcomes. This allows the level of achievement of program outcomes on a course basis to be determined. The invention differs in that it enables the determination of the level of achievement of program outcomes not only on a course basis but also on a departmental and faculty level. Additionally, this method allows for the monitoring of graduate competencies. Unlike previous techniques, it can measure a course's credit (ECTS) load based on individual students. The system allows for a comparison between the credit assigned to a course in the curriculum and the credits measured based on the time a student allocates to the course. This enables the actual credit value of a course, as perceived by the student, to be determined, allowing departments to restructure curricula in light of this information. Furthermore, while prior art only supports obtaining learning outcomes for online courses, the invention's system and method also assess the impact of face-to-face exams on learning outcomes, thereby determining program outcome performance. For face-to-face exams, students' question-based scores recorded on optical forms are processed and transferred to a digital environment via optical readers.

[0068] In the prior art, performance measurement is conducted based on learning outcomes, and academic performance evaluation is also carried out. However, unlike the prior art, the system and method of the invention link program outcomes with learning outcomes, enabling access to these outcomes both at the course level and at the unit level in light of all results. This approach not only provides performance evaluation but also facilitates the measurement of graduate competence at both unit and faculty levels. Graduate competence, an essential aspect of accreditation requirements, reflects the ability to meet program outcomes. This online program design addresses such parameters effectively. The prior art does not provide a solution for such parameters. While each course includes learning outcomes, the level of attainment of program outcomes and their associated objectives are generally absent in foreign patents in the prior art.

[0069] The system and method of the invention differ from the prior art by being able to measure the credit (ECTS) load of a course based on the individual student. It enables a comparison between the credit value of a course in the curriculum and the credit measured by the time the student dedicates to the course. This allows the actual credit value of the course, as perceived by the student, to be determined, and department curricula can be structured based on this information. With the system and method of the invention, course credits can be calculated based on the time students allocate to the course, allowing a comparison between the curriculum-defined course load and the actual course loads. This enables departmental curricula to be adjusted according to the actual credits, and new courses can be added or removed from the curriculum if deemed necessary. Additionally, instructors can be evaluated through student surveys, and instructor performance based on student feedback can be measured using the Instructor Performance System (140).

[0070] The server of the system subject to the invention facilitates the execution of the following steps.

[0071] • Entry of course scores into the system on a question-by-question basis,

[0072] • Determining the level of achievement of learning outcomes (LOs) by entering the contribution of each question to LOs into the system,

[0073] • Calculating the level of achievement of program outcomes (POs) by retrieving the LO- PO relationships through the Bologna system,

[0074] • Displaying the level of achievement of LOs and POs on a student basis,

[0075] • Determining the level of achievement of LOs on a course basis,

[0076] • Determining the level of achievement of POs on a program or department basis,

[0077] • Identifying the extent to which each course impacts the POs,

[0078] • Communicating with different systems used by the university, such as the grading system, Bologna information package, and quality management system,

[0079] • Allowing instructors to upload their course description files onto the system,

[0080] • Enabling instructors to submit continuous improvement suggestions through the system,

[0081] • Reporting the performance scores of all instructors within a department,

[0082] • Assigning an educational committee through the system to review and score the performed activities,

[0083] • Determining the performance score of an instructor by evaluating LO achievement levels, course description files, continuous improvement suggestions, and LO-PO matrix alignment, • Automatically transferring grades entered into the grading system to the performance evaluation system,

[0084] • Redirecting continuous improvement suggestions as Corrective and Preventive Actions (CPAs) to the quality management system,

[0085] • Displaying the overall performance status of all programs on the main dashboard,

[0086] • Automatically pulling the results of student surveys into the system and reflecting survey scores on the instructor's performance,

[0087] • T ransferring the results of ECTS workload surveys into the system to compare the calculated ECTS with the actual ECTS of the course.

[0088] The system and method of the invention allow course credits to be calculated based on the time students allocate to the course, enabling a comparison between the curriculum's assigned course load and the actual course load. This allows department curricula to be adjusted according to actual credits, and new courses can be added or removed from the curriculum as necessary. Additionally, with the student survey module (150), instructors can be evaluated, and instructor performance based on student feedback is measured using the instructor performance system (140). The system and method of the invention calculate the level of achievement of learning outcomes by processing exam results into the system on a question-by-question basis. Using the results of these calculations, the system determines the course's contribution to program outcomes and evaluates the student's level of achievement of program outcomes both periodically and at graduation. In Figure 2, the workflow for displaying the success of Learning Outcomes (LOs) in the relevant exam is provided. First, exam scores are entered (1000). Then, the success status of the questions is determined (1001 ). Subsequently, the exam LO success is identified (1002), and the Question x LO relationship is calculated (1003).

[0089] In Figure 3, the workflow for determining the average contribution of a course to program outcomes (POs) is illustrated by summing each column of results and dividing by the number of students. The workflow includes obtaining the LO*PO matrix from the Bologna information package (1004). Subsequently, the course PO success is entered (1005), followed by the course LO success (1006). Multiple exam LO successes are considered for the course LO success. For instance, values such as Exam1 LO success (1007), Exam2 LO success (1008), and Exam(n) LO success (1009) are included, where n represents the number of values.

[0090] In Figure 4, the workflow for determining the level of achievement of program outcomes (POs) is presented. Here, the contribution level of all courses to the program outcomes is calculated by multiplying the PO contribution of each course by the ratio of the ECTS load of the course to the total ECTS load and then summing these values. The ECTS loads of the courses are retrieved from the Bologna information package (1010). Subsequently, the program PO success is entered (101 1 ). For the program PO success, multiple course PO successes are considered. For example, values such as Coursel PO success (1012), Course2 PO success (1013), and Course(n) PO success (1014) are included, where n represents the number of values.

[0091] The system described in the invention consists of two modules, which operate on a computer or server. The first module is the "Course Introduction" module, and the second module is the "Assessment and Evaluation" module (Figure 1 ). The Course Introduction module contains course introduction files uploaded by the course instructor. The system evaluates the accuracy and completeness of these uploaded files, and the results are reflected in the instructor's performance score.

[0092] The "Assessment and Evaluation" module is designed for instructors to input exam scores related to their courses into the system. Instructors can manually enter student scores for each question, or the system can retrieve question-based scores from the university's student grading system. Another data input for this module involves the relationship between exam questions and the corresponding learning outcomes. The instructor completes the data entry process by entering the question-LO relationship into the system.

[0093] A system related to measurement and evaluation has been added to the education and training performance system. Below are explanations of the system's operations, calculations, and evaluations outlined in bullet points:

[0094] • In the system, the instructor can log in to eopd.omu.edu.tr using their username and password to view the courses they are teaching. • The instructor must select the courses for which they want to perform the measurement and evaluation and determine the contribution (%) of the evaluation tools (midterm, quiz, final, etc.) to the final grade, ensuring the total is 100.

[0095] • The learning outcomes (LOs) of each course the instructor is responsible for must be up-to-date and matched with the LO-PO matrix. This information is automatically retrieved from ebs.omu.edu.tr. After data entry is completed, the system performs the following calculations and displays the levels of attainment for learning and program outcomes.

[0096] • The calculations performed by the system are summarized below:

[0097] ❖ On the grades page of the system, students' ID numbers and the scores they received for each question are listed. The relationship between the questions and the LOs must be entered on another page.

[0098] ❖ The scores that each student received for the exam questions are entered into the grades section corresponding to the question numbers. Additionally, these scores can be automatically retrieved from the university's course grade system.

[0099] ❖ A Grades Matrix is created, consisting of the ratio of the scores obtained for each question to the maximum possible score for that question, covering all students who took the exam. An example matrix is provided below (Equation 1 ). Each row in the matrix represents a student, and each column represents a question.

[0100] *> A Learning Outcome Matrix (SxLO - Equation 2) is created, in which exam questions are linked to the corresponding learning outcomes. This matrix consists of Os and 1 s. If a question is related to a learning outcome, the intersecting cell of the question and the learning outcome is assigned a value of 1 . If the question is not related to a learning outcome, the value is 0.

[0101] Equation 2

[0102] ❖ Each row of the Grade Matrix represents a student. Each row of this matrix is multiplied by the Learning Outcome Matrix.

[0103] Equation 3

[0104] *> As a result of this multiplication, a matrix with dimensions corresponding to the number of students x learning outcomes (Equation 3) is obtained. Each row (vector) of this matrix represents a student. Each element of this vector is divided by the sum of the corresponding column in the SxLO matrix. For instance, the first element of the vector is divided by the sum of the first column in the SxLO matrix. This vector (Equation 4) represents the success of the learning outcomes of the respective exam (Figure 2).

[0105] Equation 4

[0106] ❖ The obtained vector is multiplied by the contribution rate of the exam and stored. For instance, if these calculations are performed for midterm grades and the midterm contributes 40%, the vector is multiplied by 0.4 and stored accordingly.

[0107] ❖ The stored data for each exam of the course (midterm, final, quiz, assignment, etc.) are aggregated to calculate and store the students' achievement rates for the course's Learning Outcomes (LO). Additionally, the success rates of all students are summed in each column and divided by the number of students to calculate the average success rate of the course's Learning Outcomes (LOA).

[0108] ❖ The LOA vector (each row of the LOA) must be multiplied by the LOxPO matrix.

[0109] ❖ Each element of the resulting new vector should be divided by the sum of each column of the LOxPO matrix.

[0110] ❖ The results of each column are summed and divided by the number of students to determine the average contribution of the course to the Program Outcomes (PO) (Figure 3).

[0111] • The results provided by the system display each student's Learning Outcome achievement for each exam.

[0112] • The system shows each student's overall success in achieving the course's Learning Outcomes.

[0113] • At the bottom of the page, the system displays the success rates of the course in achieving its Learning Outcomes.

[0114] • The contribution levels of all courses to the Program Outcomes (PO) are calculated by multiplying the ECTS weight of each course by the ratio of ECTS weight to the total ECTS weight and summing the results, thereby determining the level of achievement of the Program Outcomes (Figure 4).

[0115] • These data are stored for each student individually, and the level of achievement of Program Outcomes at graduation can be displayed for each student.

[0116] • After the calculations of LO and PO are completed, the system evaluates and scores the instructor based on their data entry status.

[0117] • Additionally, the education committee appointed by the system administrator verifies the accuracy of the entered data and assigns a separate score.

[0118] • Furthermore, survey results from the university's other modules are reflected in the respective instructor's evaluation, contributing to their survey-based score.

[0119] • The system combines all these scores by multiplying them with weights set by the system administrator to calculate the instructor's performance score.

[0120] • Additionally, the system allows instructors to submit continuous improvement suggestions directly to the Quality Management System (160), enabling automatic creation of Corrective and Preventive Action (DIF) requests.

[0121] In the invention, the modules considered the primary technical superiority begin with faculty members entering course-specific and question-specific data into the system under the Calculations module. Subsequently, matrix connections are established between Learning Outcomes (LOs) and Program Outcomes (PCs). As a result, the level of achievement of program outcomes at both the course and departmental levels can be viewed as system outputs. This module also enables faculty members to plan continuous improvements on a course basis. When implemented within a university, this system allows the determination of graduate competency levels, which are essential for accreditation processes within the institution. The system comprises Calculations, Learning Outcomes, Program Outcomes, the Student Survey Module (150), and Continuous Improvement Modules. Interconnections are established between these various modules. Learning Outcomes and Program Outcomes are displayed to the user via the interface.

[0122] The level of students' achievement in learning and program outcomes can be measured independently of their exam grades, providing a more comprehensive and detailed evaluation result. Additionally, the system includes an ECTS measurement module that evaluates the students' workload while achieving program outcomes and calculates the ECTS credits for the course accordingly. This enables a comparison between the curriculum's assigned ECTS credits and the actual ECTS achieved by students. By evaluating the level of attainment of program outcomes together with the ECTS credits assigned to the courses in the program, both student and instructor performance can be assessed. Accordingly, the ECTS credits of the courses in the curriculum can be reorganized, and new courses can be added to the curriculum as necessary.

[0123] The system and method of the invention provide advantages in the following aspects:

[0124] Inputting Course Scores on a Question Basis: The invention enables course scores to be entered into the system on a question-by-question basis, allowing for the determination of each question's contribution to students' learning outcomes. This facilitates a more detailed assessment of the level of achievement of learning outcomes.

[0125] Extracting the LO-PO Relationship Through the Bologna System: The invention allows for the extraction of the relationship between learning outcomes (LO) and program outcomes (PO) through the Bologna system, enabling the calculation of the level of achievement of program outcomes. This assists in evaluating the compliance of educational programs with the standards of the European Higher Education Area.

[0126] Displaying Achievement Levels for Students and Courses: The invention allows for the display of achievement levels for learning outcomes both on a student and course basis. This enables a detailed analysis of the performance of students and courses.

[0127] Determining Achievement Levels for Program or Department-Based Program Outcomes: The invention can determine the achievement levels for program outcomes across an entire program or department. This allows for the evaluation of the overall success of educational programs.

[0128] Communication with Different Systems: The invention can communicate with various systems used by the university, such as the grade recording system, Bologna information package, and quality management system (160). This integration facilitates data flow and streamlines processing procedures.

[0129] Continuous Improvement Suggestions and Performance Reporting for Instructors: The invention enables instructors to upload course information files, provide continuous improvement suggestions, and have their performance reported. Additionally, through the education commission system (130), an education commission can be appointed to oversee and evaluate the performed tasks.

[0130] Detailed Technical Explanation and Modules: The invention consists of two main modules: the "Course Introduction" module and the "Assessment and Evaluation" module. The Course Introduction module includes course information files, while the Assessment and Evaluation module allows instructors to input exam scores and the question-learning outcome relationships into the system. Addition of an Assessment and Evaluation System: The invention incorporates a system for assessment and evaluation into the education and training performance system. This system enables instructors to evaluate their courses, update learning outcomes, and calculate the attainment levels of both learning outcomes and program outcomes.

[0131] REFERENCES

[0132] Kirkavak, N„ Ozakta§, H. & Ertem, M. A. (2019). PROGRAM QIKTILARI iQiN ALTERNATiF BiR OLQME-DEGERLENDiRME MODELi: MUDEK KARNESl. Endustri Muhendisligi, 30 (1 ), 32-48. https: / / derqiDark.orq.tr / tr / pub / endustrimuhendisliqi / issue / 48367 / 601449

Claims

CLAIMS1. A system that indicates the level of achievement of students in program outcomes, evaluates the teaching performance of instructors in a course, and communicates with course information packages, characterized by: A course introduction module (110) that includes course description files uploaded by the instructor, ensuring that the files are correctly uploaded and complete, with the system scoring them and reflecting the results in the instructor's performance evaluation; An assessment and evaluation module (120) that allows instructors to input exam scores for their courses into the system, manually enter question-based scores for students, or automatically retrieve question-based scores from the university's student grading system, and enables the instructor to associate exam questions with learning outcomes, completing the data entry process; A computation module that calculates the level of achievement of students in learning outcomes based on question-based exam results entered into the system, and, using these calculations, determines the contribution of the course to program outcomes and the level of achievement of program outcomes for students on a semester and graduation basis; A student survey module (150) for evaluating instructors and measuring instructor performance based on student feedback; A continuous improvement module that allows instructors to upload course description files, submit continuous improvement suggestions, and have their performance reported; A server that integrates the aforementioned modules; An interface for displaying learning outcomes and program outcomes; A course introduction database (1 11 ) providing data for the course introduction module; and An assessment and evaluation database (121 ) providing data for the assessment and evaluation module.2 The system according to Claim 1 , characterized by an assessment and evaluation module that allows data entry of the relationship between exam questions and learning outcomes defined by the instructor.3 The system according to Claim 1 , characterized by:• Entering course scores into the system on a question-by-question basis,• Determining the level of achievement of learning outcomes (LOs) by entering the contribution of each question to LOs into the system,• Calculating the level of achievement of program outcomes (POs) by retrieving the LO-POrelationships through the Bologna system,• Displaying the level of achievement of LOs and POs on a student basis,• Determining the level of achievement of LOs on a course basis,• Determining the level of achievement of POs on a program or department basis,• Communicating with the university's grading system, Bologna information package, and quality management system (160),• Allowing instructors to upload their course description files onto the system,• Enabling instructors to submit continuous improvement suggestions through the system,• Reporting the performance scores of all instructors within a department,• Assigning an educational committee through the system (130) to review and score the performed activities,• Determining the performance score of an instructor by evaluating LO achievement levels, course description files, continuous improvement suggestions, and LO-PO matrix alignment,• Automatically transferring grades entered into the grading system to the performance evaluation system,• Redirecting continuous improvement suggestions as Corrective and Preventive Actions (CPAs) to the quality management system (160),• Displaying the overall performance status of all programs on the interface,• Automatically pulling the results of student surveys into the system and reflecting survey scores on the instructor's performance,• Transferring the results of ECTS workload surveys into the system to compare the calculated ECTS with the actual ECTS of the course,• Displaying the levels of attainment of learning and program outcomes on the interface using the data retrieved from the course introduction database (11 1 ) and the assessment and evaluation database (121 ).4 The system according to Claim 1 , characterized by a computation module that calculates the level of achievement of program outcomes (POs) by retrieving the relationship between learning outcomes (LOs) and program outcomes (POs) through the Bologna system.5 The system according to Claim 1 , characterized by an interface that allows the level of achievement of learning outcomes (LOs) to be displayed both on a student and course basis.6 The system according to Claim 1 , characterized by a server that communicates with the university's grading system, Bologna information package, and quality management system (160).7 The system according to Claim 1 , characterized by:• Inputting the relationship between exam questions and learning outcomes (LOs),• Entering the question-based scores obtained by each student for exam questions into the appropriate question numbers or automatically retrieving them from the university's grading system,• Creating a Grade Matrix consisting of the ratio of question-based scores to the maximum possible score, covering all students who took the exam,• Creating a Learning Outcome Matrix (SxLO) that associates exam questions with the corresponding learning outcomes,• Multiplying each row of the Grade Matrix, representing a student, with the Learning Outcome Matrix,• Generating a matrix with dimensions of the number of students x learning outcomes as a result of this multiplication,• Dividing each element of the resulting matrix by the sum of the corresponding column in the SxLO matrix,• Multiplying the obtained vector by the contribution rate of the exam and storing the results,• Aggregating the stored data for all exams in a course to calculate and store the achievement rates of the course's LOs,• Summing the success rates of all students in each column and dividing by the number of students to calculate the average success rate of the course's LOs (LOA),• Multiplying each row of the LOA vector by the LOxPO matrix,• Dividing each element of the resulting vector by the sum of the corresponding column of the LOxPO matrix,• Summing the results of each column and dividing by the number of students to determine the average contribution of the course to program outcomes (POs).

Images