Artificially intelligent assessment system

Abstract

An artificially intelligent assessment system including a language model (LM) unit adapted to generate content assessment output based on content input and content assessment rubric input, further including a Diffusion Model (DM) unit adapted to generate content assessment output imagery, the DM unit being in operative communication with the LM unit.

Description

[0001] ARTIFICIALLY INTELLIGENT ASSESSMENT SYSTEM

[0002] The present invention relates to automation and in particular to automated assessment of textual and imagery content.

[0003] The invention has been developed primarily as an artificially intelligent assessment system for automatically assessing the correctness of textual and imagery content against a rubric by means of Artificial Intelligence and will be described herein with reference to this application. However, it will be appreciated that the invention is not limited to this particular field.

[0004] BACKGROUND OF THE INVENTION

[0005] Any discussion of prior art throughout the specification should in no way be considered as an admission that such prior art is widely known or forms part of the common general knowledge in the field.

[0006] Assessing written and imagery content (i.e., textual content, text, etc.) by using a rubric involves systematically comparing the attributes of the content against predetermined criteria outlined in the rubric. The process necessitates a thorough examination of the text to determine the extent to which it meets or deviates from the criteria defined in the rubric. Based on this comparison, feedback or scores are given to reflect the content's alignment with the rubric's criteria.

[0007] Human assessments of written content can vary due to factors such as fatigue, biases, or individual interpretation. The consistency of evaluations also may fluctuate over time, even with a well-defined rubric. Personal experiences or inherent beliefs of the assessor may inadvertently influence judgments, leading to potential inconsistencies in evaluations across similar content. Human assessments are typically time-intensive, especially with dense or complex texts.

[0008] Known software tools commonly utilized in the assessment of written content are annotation tools, Learning Management Systems (LMS), and collaboration platforms. Annotation software tools facilitate direct commentary on specific textual portions, but their overuse can lead to cluttered content. Learning Management Systems (LMS) offer organized environments for content and feedback, but their extensive features can introduce unnecessary complexity. Collaboration platforms enable input from multiple assessors or peer reviews; however, they can pose challenges in reconciling diverse feedback and ensuring consistent evaluations.

[0009] OBJECT OF THE INVENTION

[0010] It is an object of the invention to overcome or ameliorate at least one of the disadvantages of the prior art, or to provide a useful alternative.

[0011] It is an object of the invention in its preferred form to provide an artificially intelligent system that reduces the time taken to assess textual and imagery content, provides more consistent assessments, reduces biases, and is straightforward to use.

[0012] SUMMARY OF THE INVENTION

[0013] According to the invention there is provided an artificially intelligent assessment system including: a language model (LM) unit adapted to generate content assessment output based on content input and content assessment rubric input.

[0014] Preferably, there is further included a Diffusion Model (DM) unit adapted to generate content assessment output imagery, the DM unit being in operative communication with the LM unit.

[0015] Preferably, there is further included a prompt unit in operative communication with the LM unit, the prompt unit being adapted to receive content input and content assessment rubric input.

[0016] Preferably, there is further included a Reinforcement Learning (RL) unit in operative communication with the LM unit and / or DM unit adapted to train the LM and / or the DM according to one or more policies.

[0017] Preferably, there is further included an automatic fine-tuning unit adapted to configure the LM and / or the DM to generate a uniquely characterized assessment output Preferably, there is further included an auxiliary context unit in operative communication with the LM unit, the auxiliary context unit being adapted to provide content assessment contextual information.

[0018] Preferably, there is further included a document processor unit in operative communication with the prompt unit, the document processor unit being adapted to receive and process documents.

[0019] Preferably, there is further included an Optical Character Recognition (OCR) unit in communication with the LM and / or DM unit, the OCR unit adapted to generate OCR-processed output

[0020] The content assessment output is preferably text.

[0021] Preferably there is further included a data generation unit in operative communication with the LM unit and / or DM unit, the data generation unit being adapted to generate LM unit and / or DM unit input.

[0022] The data generation unit is preferably in operative communication with the RLunit instead of directly or indirectly with the LM unit and / or the DM unit.

[0023] The automatic fine-tuning unit preferably uses Low-Rank Adaptation (LoRa) to configure the LM unit and / or the DM unit to generate uniquely characterised assessment output.

[0024] Preferably, there is further included an evaluation unit adapted to evaluate the performance of uniquely characterised assessment output.

[0025] The RL unit is preferably in operative communication with the automatic fine-tuning unit and the document processor unit instead of directly or indirectly with the LM unit and / or DM unit.

[0026] The automatic fine-tuning unit is preferably in operative communication with the document processor unit and the evaluation unit instead of directly or indirectly with the LM unit and / or a DM unit.

[0027] The auxiliary context unit preferably further includes a semantic chunk unit and a vector database, the vector database being adapted to store content assessment contextual information embeddings receivable from the semantic chunk unit, the semantic chunk unit being adapted to generate assessment contextual information embeddings from contextual information text receivable by the semantic chunk unit.

[0028] The auxiliary context unit is preferably in operative communication with the prompt unit instead of directly or indirectly with the LM unit.

[0029] The OCR unit is preferably in communication with the document processor unit instead of directly or indirectly with the LM and / or a DM unit.

[0030] One or more uniquely characterised assessment outputs is preferably combined with another one or more assessment outputs.

[0031] Preferably, there is further included n databases in operative communication with one or more units, the n databases being adapted to store data corresponding to any one or more units.

[0032] Preferably, there is further included one or more of, a GUI, an input / output document interface, a RL interface and a document generator interface adapted to generate documents from LM unit and / or DM unit output.

[0033] BRIEF DESCRIPTION OF THE DRAWINGS

[0034] A preferred embodiment of the invention will now be described, by way of example only, with reference to the accompanying drawings in which:

[0035] Figure 1 is a block diagram according to the invention;

[0036] Figure 2 is an image showing input content and content assessment output text;

[0037] Figure 3 is an image showing output text; and

[0038] Figure 4 is an image showing output imagery and text.

[0039] PREFERRED EMBODIMENT OF THE INVENTION

[0040] Referring to the drawings, Figure 1 shows the artificially intelligent assessment system 1 has a

[0041] Language Model (LM) 2 unit adapted to generate content assessment output in the form of text 4, and a Diffusion Model (DM) unit 6 adapted to generate content assessment output imagery 8. The DM unit 6 is in operative communication 10 with the LM unit 2. The content assessment output text 4 and content assessment output imagery 8 are generated by the LM unit 2 and / or correspondingly the DM unit 6 based on content assessment input and content assessment rubric input in the form of text and / or imagery 12. The content assessment output text 4 and content assessment output imagery 8 can contain input content 12A along with content assessment output text 4A, just the content assessment output text 4A, and in the case of content assessment output imagery 8 can also contain handwriting and / or images 8A, or another suitable combination (not shown), as best seen in Figure 2.

[0042] A prompt unit 14 adapted to receive content input and content assessment rubric input 12 is in operative communication with the LM unit 2 and / or the DM unit 6. The prompt unit 14 is adapted to prompt the LM unit 2 and / or the DM unit 4 with a prompt (not shown) containing content input and content assessment rubric input 12 and thus trigger the LM unit 2 and / or the DM unit 6 to generate corresponding content assessment output text 4 and / or content assessment output imagery 8.

[0043] The prompt unit 14 is also adapted to determine if additional assessment contextual information 16 is required by the LM unit 2 and / or DM unit 6 to generate content assessment output 4 and / or 6. The prompt unit 14 is adapted to determine if further assessment contextual information 16 is required by means of hard coded rules, and / or user preferences, and / or Agent Self-Reflection (ASR).

[0044] Hard coded rules define baseline criteria for information sufficiency based on pre-determined assessment scenarios. In a preferred embodiment, hard coded rules require that at least content input reference text related to the content input, and / or at least one subject-specific content assessment rubric input, and / or at least one generic example of content assessment output text and / or content assessment output imagery will be available to the prompt unit. In another preferred embodiment, the hard coded rules can relate to any suitable aspect of making assessments.

[0045] User preferences can be adjusted by the user (not shown). In a preferred embodiment, the user can select their preferences in the form of content input reference text, subject-specific content assessment rubric input, and / or generic examples of content assessment text and / or content assessment output imagery. In another preferred embodiment, user preferences can include assessment styles, linguistic patterns, and handwriting characteristics. In yet another preferred embodiment, user preferences can relate to any suitable aspect of making assessments.

[0046] ASR dynamically analyses a prompt and its informational context, relative to the LM and / or DM, to evaluate the relevance and completeness of information, determines if additional information is required by assessing the alignment of the prompt and content assessment rubric input, and reorganises and / or regenerates a new prompt if so required.

[0047] Further, the system 1 has a Reinforcement Learning (RL) unit 18, an automatic fine-tuning unit 20, a document processor unit 22, and an evaluation unit 24. The RL unit 18 is operative communication 10 with the fine-tuning unit 20 and the document processor unit 22. In turn, the document processor unit 22 is in operative communication with the prompt unit 14.

[0048] The RL unit 18 uses RL in the form of Reinforcement Learning from Human Feedback (RLHF) architecture 26 having a supervised feedback interface whereby a user reinforces initial target RL policy or policies by means of Supervised Learning. When sufficiently reinforced by the user, the target RL policy or policies continue to be reinforced by means of non-supervised RL and then are further used to train and / or configure the LM unit 2 and / or the DM unit 6 by means of an optimisation algorithm such as Proximal Policy Optimisation (PPO) (not shown).

[0049] The LM unit 2 and / or the DM unit 6 each have corresponding LoRA units 28A and 28B in the form of adapters 30A and 30B corresponding to n user IDs (not shown). The automatic fine-tuning unit 20 is adapted to configure the LM unit 2 and / or the DM unit 6 by means of the LoRA units 28A and 28B to generate content assessment output and / or content assessment output imagery in the form of uniquely characterised assessment outputs (i.e., content assessment output text and / or content assessment output imagery 4 and 8) using Low-Rank Adaptation (LoRA). The automatic fine-tuning unit 20 is adapted to configure the LM and / or the DM unit to generate uniquely characterised assessment outputs using supervised learning, whereby a user having with a corresponding ID applies labels to data used to configure corresponding LoRA adaptors 30A and 30B. Additionally, the automatic fine-tuning unit 20 further includes Direct Preference Optimisation (DPO) (not shown) either in parallel or serially with the LoRA units 28A and 28B to generate n uniquely characterised assessment outputs. Advantageously, DPO captures aspects corresponding to uniquely characterised assessment outputs that can be difficult to capture using supervised learning. In a preferred embodiment, uniquely characterised assessment outputs correspond to assessment styles, linguistic patterns, and handwriting characteristics.

[0050] The automatic fine-tuning unit 20 can be configured to combine and / or aggregate one or more uniquely characterised assessment outputs with another one or more combined assessment outputs to advantageously leverage multiple uniquely characterised assessment outputs. In a preferred embodiment, the uniquely characterised assessment outputs are combined to produce crossteacher assessment, comparative analysis, federated assessment (not shown). In another preferred embodiment, n LoRA units 30A and 30B can be selected and combined by the user to generate desired uniquely characterised assessment outputs (i.e., 4 and 6). The automatic fine-tuning unit 20 is in operative communication 10 with the document processor unit 22 and the evaluation unit 25.

[0051] The document processor unit 22 is adapted to receive and process documents (i.e., input 12 and / or information 16) from one type to another type suitable for the system 1, such as, for example, PDF to TXT.

[0052] The evaluation unit 24 is adapted to evaluate the performance of the uniquely characterised assessment output by means of user generated corrections and supervised learning whereby a user applies labels to uniquely characterised assessment output.

[0053] The system 1 also has an auxiliary context unit 32 in operative communication 10 with the prompt unit 14. The auxiliary content unit 32 has a semantic chunk unit 34 and a vector database 36 in operative communication with one another 10 and adapted to store content assessment contextual information embeddings (not shown) receivable from the semantic chunk unit. The semantic chunk unit 34 is adapted to generate assessment contextual information embeddings from contextual information 16 (i.e., text, images, etc.) receivable by the semantic chunk unit 34 from user input and / or automated input resulting from information searches such as, for example, an internet search. In a preferred embodiment, auxiliary context unit 32 can query another suitable unit adapted to provide services such as OCR.

[0054] A data generation unit 36 is in operative communication 10 with the RL unit 18. The data generation unit 36 is adapted to generate LM unit 2 and / or DM unit 8 input in the form of content assessment input and content assessment rubric input 12 and / or uniquely characterised assessment outputs, thus advantageously providing additional artificially generated training data to facilitate LM unit 2 and / or DM unit 6 performance improvements.

[0055] An Optical Character Recognition (OCR) unit 38 is in operative communication with the document processor unit 22. The OCR unit is adapted to generate OCR-processed output based on inputs received from the document processor unit such as image files containing writing and / or drawings, text files, and other target inputs. The inputs being and processed outputs being OCR-processed content assessment input and content assessment rubric input (i.e., content assessment input and content assessment rubric input).

[0056] The system 1 has n databases in the form of three databases: a data bucket 40A, a LoRa bucket 40B and an output database 40C. The databases 40A, 40B, and 40C, are adapted to store data to a corresponding unit with which it is in operative communication 10 with. In the present embodiment, the data bucket 40A is in operative communication with the document processor unit 22, the RL unit 18, and the automatic fine-tuning unit 20. The LoRa bucket 40B is in operative communication with the LM unit 2 LoRA unit 28A and the DM unit 6 LoRA unit 28B as well as the evaluation unit 24. The output database 40C is in communication with the LM unit 2, the DM unit 6 and data bucket 40A.

[0057] The system has a GUI 42 adapted to facilitate interaction with the system 1 and the units comprising the system 1, such as an input / output document interface, an RL interface, and a document generator interface adapted to generate documents from LM unit and DM unit output. In a preferred embodiment, the GUI is adapted to interface with n units and n system features.

[0058] To use the artificially intelligent assessment system 1, the user (not shown) provides documents (or i.e., a single document, etc.) containing target content input 12 and target content assessment rubric input 12 to the document processor unit 22 using the 42 GUI and the input / output document interface along with an ID number corresponding to the user. Depending on the document type, if required, the document processor unit 22 then sends the documents as inputs to the OCR unit 38 for processing. The OCR unit 38 generates OCR-processed output in the form content input and content assessment rubric input text (i.e., text data etc.) recognised by the OCR unit 38 within the input documents 12, and then sends the content input and content assessment rubric text data to the prompt unit 14. The prompt unit 14 determines if further context is required for the LM unit 2 to assess the target content input 12 against the target content assessment rubric input 12 by checking against hard coded rules, and / or user preferences, and / or ASR. Using the GUI 42, the user can select their preferences at this or an earlier step.

[0059] If further context is required, the prompt unit 14 queries the auxiliary context unit 32 vector database 36 for content assessment contextual information stored in the vector database that has been received from the semantic chunk unit 34. If no suitable assessment contextual information is found, then, buy using the GUI 42 and the input / output document interface the user can submit further information 16 in the form of documents to the semantic chunk unit 34. The semantic chunk unit 34 then generates assessment contextual embeddings by processing the information 16 documents input by the user and stores the assessment contextual embeddings in the vector database 36. When the prompt unit 14 subsequently queries the auxiliary context unit 32 vector database 36 for content assessment contextual information the auxiliary context unit 32 sends the content assessment contextual information to the prompt unit 14.

[0060] Next, the content input 12 and content assessment input 12, the user ID, and content assessment contextual information are sent as a prompt to the LM unit 2. The LM unit 2 then generates content assessment output text 4 based on the content input 12, content assessment rubric input 12, and adaptor 30A corresponding to the user ID. The output text 4 can be stored in the database 40C.

[0061] The GUI 42 and document generator interface can be used to generate a document readable by the user, containing an assessment of the target input content 12 based on the content assessment rubric input 12. Similarly, the user can also use the GUI 42 to generate content assessment output imagery 8 instead of, or together with, the content assessment output text 4.

[0062] The content assessment output text 4 and content assessment output imagery 8 can contain input content 12A along with content assessment output text 4A, just the content assessment output text 4B, and in the case of content assessment output imagery 8 can also contain handwriting 8A and / or images 8B, or another suitable combination (not shown), as best seen in Figure 2, 3, and 4.

[0063] In a preferred embodiment, the user accesses the automatic fine-tuning unit 20 by using the GUI 42 and configures the unit 20 to combine and / or aggregate one or more uniquely characterised assessment outputs with another one or more uniquely characterised assessment outputs. In a preferred embodiment, the user accesses the automatic fine-tuning unit 20 by using the GUI 42 and configures the unit 20 to produce cross-teacher assessment, comparative analysis, and federated assessment.

[0064] In a preferred embodiment, the user accesses the automatic fine-tuning unit 20 by using the GUI 42 and configures the unit 20 to select n LoRA 30A and / or 30B units to be selected and combined by the user to generate desired uniquely characterised assessment outputs.

[0065] The performance of the LM unit 2 and / or the DM unit 6 can be improved by the user accessing the GUI 42 and RL interface to provide RL unit 18 RHLF-based 26 Supervised Learning feedback to reinforce a target RL policy or policies by labelling "good" or "bad" the content assessment output text 4 and content assessment output imagery 8 generated by the LM unit 2 and / or DM unit 6 until the policy, or policies are sufficiently accurate (i.e., sufficiently reinforced) at producing "good" content assessment output text 4 and content assessment output imagery 8. The resulting reinforced policy or policies can then be used to further train and thus improve the performance of the LM unit 2 and / or DM unit 8. In a preferred embodiment, proximal policy optimisation is used to further train the LM unit 2 and / or DM unit 8. In a preferred embodiment, another suitable type of optimisation algorithm can be used.

[0066] Next, evaluation unit 24 then evaluates the performance of the uniquely characterised assessment (i.e., output text and output imagery) and is accepted or rejected.

[0067] Additionally, the data generation unit 36 can be accessed by the GUI 42 and the RL interface to generate content assessment input 12 and content assessment input 12 to provide additional training inputs for the RL unit 18. In a preferred embodiment, an optimisation algorithm in the form of DPO is applied to the target RL policy or policies before the LM unit 2 and / or RM unit 8 are further trained.

[0068] It will be appreciated that the illustrated artificially intelligent system 1 reduces the time taken to assess textual content, provides more consistent assessments, reduces biases, and is straightforward to use. The PSA will appreciate that computer systems ("computers") come in various forms adapted to many different uses. Mobile devices, personal computers ("PC"), clients / servers, embedded systems, industrial Programmable Logic Controllers ("PLC"), distributed systems, pervasive computers, quantum computers are all different types of computers.

[0069] Computer systems execute instructions to perform a variety of tasks. Computer systems are electrically powered devices having one or more processing units for executing instructions in the form of software programs. Central Processing Units ("CPU"), Graphics Processing Units ("GPU"), Field Programmable Gate Arrays ("FPGAs"), Application Specific Integrated Circuits ("ASICs"), and logic circuits, are all different types of processing units. CPUs are packaged into integrated circuits ("IC") known as microprocessors, microcontrollers, or systems-on-chips ("SoC") and are typically further integrated into printed circuit boards ("PCB") to extend their functionality. Unlike microprocessors, microcontrollers and SoCs have additional peripheral circuits ("peripherals") integrated in a single circuit. Analog-to-digital converters ("ADC"), digital-to-analogue ("DAC"), timers, counters, communication interfaces, Pulse Width Modulation generators ("PWM"), General Input and Output ("GPIO"), memory, power management, are all different types of peripherals. Although computer systems can be adapted and configured in many ways, all computer systems function in essentially the same way.

[0070] The different parts of a computer system are connected by various types of electronic circuits. Circuits used in computer systems are typically manufactured into Printed Circuit Boards ("PCB") but also use wired and wireless connections. Typical electronic circuits include filter circuits, amplification circuits, power circuits, communication circuits, safety circuits, and switching circuits. Circuits are selected or designed according to well-known principles to be compatible with the computer system and enable it to operate. Such circuits define operative connections including operative communication connections.

[0071] Computer systems commonly have memory in the form of volatile memory such as random-access memory ("RAM") or non-volatile memory (e.g., Hard Disk or others) and input and output terminals, all of which are typically accessed by a processor while the computer system is operating. Hardware peripherals such as keyboards, mice, audio and video cards, printers, sensors, actuators, communication interfaces, and others, are often operatively connected to the input and output terminals of a computer system to further enhance its functionality and usability. Processing units access software program instructions from memory. Software program instructions are typically organised in a main program that calls various functions or methods which in the present invention in preferred embodiments are referred to as "units". Software program instructions processed by a processor from machine-readable code that that has been typically translated from a higher-level computer language such as C, C++, Java, Python, or from low level assembly language or binary code. Software programs typically use Application Program Interfaces ("APIs") to communicate (i.e., operatively communicate) between one part of a program (i.e., unit) and another, as well as with other programs sometimes located in another computer system. Operative communication typically involves calling functions and / or sharing variables, passing data / information. Software programs are typically designed with a top-down approach, where an increasingly higher level of abstraction is iteratively reduced in abstraction into code.

[0072] A software program can be considered as an algorithm typically formed from many sub-algorithms of varying complexity. Three main software algorithms are known as the or, while, and do-while loops. Computer algorithms also typically use Boolean Algebra. Algorithms can be used to solve mathematical, modelling, graphics, as well as other problems. Commonly used algorithms order numbers from smallest to greatest, determine the frequency of certain data values in a database, or find the greatest common divider of a set of numbers. Artificial Intelligence ("Al") typically uses complex computing, mathematical, and statistical algorithms to perform challenging tasks. One type of Al is Machine Learning ("ML").

[0073] A software program will typically include a user interface ("Ul") to allow a user to interact with the computer system by providing inputs by means of a hardware peripheral (e.g., a keyboard) to the computer system and receiving outputs from the computer system by means of another hardware peripheral (e.g., a screen). The Ul could be a Graphics Ul ("GUI").

[0074] Communication between computer systems is implemented by means of wired or wireless computer networks which use protocols to send messages back and forth between computer systems. Computer networks can take many physical and software defined forms. A pervasive type of computer network is the internet.

[0075] A computer system typically has an operating system ("OS") program to manage basic computer system functions such as access to the CPU, file systems, memory, inputs and outputs, peripherals, and other parts of the computer system. A Language Model (LM) (or i.e., a Large Language Model (LLM)) typically refers to a pre-trained Neural Network, such as one having Transformer architecture, that predicts the likelihood of words occurring in a language. A pre-trained model is typically capable of encapsulating linguistic and contextual patterns from text data.

[0076] A Diffusion Model (DM) typically consists of nodes and edges, wherein nodes represent textual or image entities, while edges describe the relationships between these. Diffusion models can propagate textual features across an image by iterating an assimilation of the properties of textual content in the image.

[0077] Reinforcement Learning (RL) is a machine learning approach where a computer agent learns to accomplish a task by engaging in trial and error with a dynamic environment. Through these interactions, the agent develops the ability to make decisions that maximize a reward metric. This process occurs autonomously, without direct human guidance or the need for explicit programming tailored to the task. The typical steps involved in implementing involves defining the environment, defining the reward, creating an agent and policy, and selecting the appropriate training algorithm. The agent is then trained and validated for performance in the given task.

[0078] Reinforcement Learning from Human Feedback (RLHF) is a technique to align an intelligent agent to human preferences. It typically involves training a reward model to represent human preferences, which can then be used to train other models through RL.

[0079] Agent Self-Reflection (ASR) is a process where the system monitors its actions and decisions, collects data during its operation, and reviews this data to evaluate performance. Typically, the system identifies any errors or areas for improvement and then adjusts its internal models or strategies accordingly. This process includes storing data and insights for future use, creating a feedback loop that supports continuous learning and adaptation.

[0080] Supervised Learning is a machine learning process where a model is trained using labelled data. Typically, the model is provided with input-output pairs, where the input data is accompanied by the correct output. During training, the model learns to map inputs to outputs by minimizing the difference between its predictions and the actual labels. This involves adjusting the model's internal parameters to improve accuracy. The trained model can then predict outputs for new, unseen inputs based on what it has learned.

[0081] Proximal Policy Optimization (PPO) is a reinforcement learning algorithm used to train policies by balancing exploration and exploitation. Typically, it works by optimizing a policy in small, controlled steps to prevent drastic updates that could destabilize the learning process. PPO uses a clipped objective function to ensure that policy updates stay within a specified range, reducing the likelihood of large deviations from the current policy. This approach maintains stability while allowing the model to improve its performance iteratively through trial and error in the learning environment.

[0082] Low-Rank Adaptation (LoRA) is a technique used in the fine-tuning of large language models to make the process more efficient. Instead of updating all the parameters of a pre-trained model, LoRA typically introduces additional low-rank matrices that are trained during the fine-tuning process. These matrices require fewer parameters to be updated, significantly reducing the computational resources needed. After fine-tuning, the low-rank matrices are typically combined with the original model, allowing the adapted model to perform specific tasks without the need to retrain the entire model. This method tends to preserve the efficiency of large models while making them adaptable to new tasks with minimal overhead.

[0083] Direct Preference Optimization (DPO) is a method used in machine learning to directly optimize a model based on user or system preferences rather than predefined labels or rewards. In this approach, the model is typically trained to maximize user satisfaction by learning from feedback, such as rankings or comparative choices, rather than traditional accuracy or error metrics. The model adjusts its outputs to align with these preferences, ensuring that the results it generates are more aligned with the desired outcomes as defined by user or system preferences. This method can typically be useful particularly useful in applications where subjective preferences are more relevant than objective correctness.

[0084] The teachings of the present invention can be implemented using suitable computer system techniques and approaches, such as those described above, which are often referred to as Information Technology ("IT"), Computer Science, and Embedded Systems development. The PSA will be able to apply the teachings of the present invention using one or more various other languages, one or more other suitable libraries, systems, Al, ML, and combinations thereof.

Claims

THE CLAIMS DEFINING THE INVENTION ARE AS FOLLOWS:

1. An artificially intelligent assessment system including: a language model (LM) unit adapted to generate content assessment output based on content input and content assessment rubric input.

2. An assessment system according to claim 1, further including a Diffusion Model (DM) unit adapted to generate content assessment output imagery, the DM unit being in operative communication with the LM unit.

3. An assessment system according to claim 1 or claim 2, further including a prompt unit in operative communication with the LM unit, the prompt unit being adapted to receive content input and content assessment rubric input.

4. An assessment system according to claim 1 and anyone of claims 2 to 3, further including: a Reinforcement Learning (RL) unit in operative communication with the LM unit when the present claim is appended to claim 1, the RL unit being adapted to train the LM according to one or more policies; and an RL unit in operative communication with the LM unit and / or DM unit when the present claim is appended to claim 2 or claim 3, the RL unit being adapted to train the LM and / or the DM according to one or more policies.

5. An assessment system according to claim 1 and anyone of claims 2 to 4, further including: an automatic fine-tuning unit adapted to configure the LM unit to generate a uniquely characterized assessment output when the present claim is appended to claim 1; and the automatic fine-tuning unit being adapted to configure the LM unit and / or DM unit when the present claim is appended to any one of claims 2 to 4.

6. An assessment system according to anyone of claims 1 to 5, further including an auxiliary context unit in operative communication with the LM unit, the auxiliary context unit being adapted to provide content assessment contextual information.

7. An assessment system according to any one of claims 3 to 6, further including a document processor unit in operative communication with the prompt unit, the document processor unit being adapted to receive and process documents.

8. An assessment system according to claim 1 and any one of claims 2 to 7, further including: an Optical Character Recognition (OCR) unit in communication with the LM unit when the present claim is appended to claim 1, the OCR unit adapted to generate OCR-processed output; and the OCR unit being in communication with the LM unit and / or the DM unit when the present claim is appended to any one of claims 2 to 7.

9. An assessment system according to any one of claims 1 to 8, wherein the content assessment output is text.

10. An assessment system according claim 1 and to anyone of claims 2 to 9, further including: a data generation unit in operative communication with the LM unit when appended to claim 1, the data generation unit being adapted to generate LM unit and / or DM unit input; and the data generation unit being in operative communication with the LM unit and / or DM unit when appended to any one of claims 2 to 9.

11. An assessment system according to claim 10, wherein the data generation unit is in operative communication with the RL unit instead of directly or indirectly with the LM unit and / or the DM unit.

12. An assessment system according any one of claims 5 to 11, wherein the automatic fine-tuning unit uses Low-Rank Adaptation (LoRa) to configure the LM unit and / orthe DM unit to generate uniquely characterised assessment output.

13. An assessment system according to claim 12, further including an evaluation unit adapted to evaluate the performance of uniquely characterised assessment output.

14. An assessment system according to any one of claims 4 to 13, wherein the RL unit is in operative communication with the automatic fine-tuning unit and the document processor unit instead of directly or indirectly with the LM unit and / or DM unit.

15. An assessment system according to any one of claims 5 to 14, wherein the automatic fine- tuning unit is in operative communication with the document processor unit and the evaluation unit instead of directly or indirectly with the LM unit and / or a DM unit.

16. An assessment system according to any one of claims 6 to 15, wherein the auxiliary context unit further includes a semantic chunk unit and a vector database, the vector database being adapted to store content assessment contextual information embeddings receivable from the semantic chunk unit, the semantic chunk unit being adapted to generate assessment contextual information embeddings from contextual information text receivable by the semantic chunk unit.

17. An assessment system according to any one of claims 6 to 16, wherein the auxiliary context unit is in operative communication with the prompt unit instead of directly or indirectly with the LM unit.

18. An assessment system according to any one of claims 8 to 17, wherein the OCR unit is in communication with the document processor unit instead of directly or indirectly with the LM and / or a DM unit.

19. An assessment system according to any one of claims 12 to 18, wherein one or more uniquely characterised assessment outputs is combined with another one or more assessment outputs.

20. An assessment system according to any one of the preceding claims, the system including n databases in operative communication with one or more units, the n databases being adapted to store data corresponding to any one or more units.

21. An assessment system according to any one of claims 12 to 20, wherein the LM unit and / or the DM unit further includes a LoRA unit adapted to configure the LM unit and / or the DM unit.

22. An assessment system further including one or more of:a GUI; an input / output document interface; and a RL interface; when the present claim is appended to claim 1, further including a document generator interface adapted to generate documents from LM unit output; and when the present claim is appended to any one of claims 1 to 18, further including a document generator interface adapted to generate documents from LM unit and / or DM unit output.

Images