System and method for domain graph control for adaptive learning proficiency assessment

Abstract

Systems and methods for an adaptive diagnostic assessment are disclosed. The systems and methods include: determining a first skill node of multiple skill nodes in a domain graph arranged as a directed acyclic graph where the first skill node includes a first estimated proficiency indication as a continuous marginal probability distribution, providing a first question to a learner, receiving an answer from the learner for the question, updating the domain graph based on the answer by: updating the estimated proficiency indication of the first skill node, and updating the another estimated proficiency indication of another skill node based on the answer and an edge connected to the first skill node, and providing a current proficiency indication associated with the updated domain graph.

Description

SYSTEM AND METHOD FOR DOMAIN GRAPH CONTROL FOR ADAPTIVELEARNING PROFICIENCY ASSESSMENTCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to U.S. Provisional Application No. 63 / 535,993, filed August 31, 2023, titled “SYSTEM AND METHOD FOR DOMAIN GRAPH CONTROL FOR ADAPTIVE LEARNING PROFICIENCY ASSESSMENT,” which is hereby incorporated by reference in its entirety.TECHNICAL FIELD

[0002] This disclosure relates to the field of systems and methods for computer-based learning assessment.SUMMARY

[0003] The disclosed technology relates to systems and methods for adaptive diagnostic assessment, including one or more server hardware computing devices or client hardware computing devices, communicatively coupled to a network, and each including at least one processor executing specific computer-executable instructions within a memory that, when executed, cause the system to: determine, by an electronic processor, a first skill node of a plurality of skill nodes in a domain graph stored in an electronic memory, the plurality of skill nodes of the domain graph arranged as a directed acyclic graph, the first skill node: comprising a first estimated proficiency indication as a continuous marginal probability distribution, being associated with one or more questions, and being connected to one or more adjacent skill nodes of the plurality of skill nodes via one or more edges, the one or more adjacent skill nodes including a second skill node that comprises a second estimated proficiency and that is connected to the first skill node via a first edge of the one or more edges; providing, by the electronic processor, a first question of the one or more questions to a learner; receiving, by the electronic processor, a first answer from the learner for the first question; updating, by the electronic processor, the domain graph based on the first answer by: updating the first estimated proficiency indication of the first skill node, and updating the second estimated proficiency indication of the second skill node based on the first answer and the first edge; and providing, by the electronic processor, a current proficiency indication associated with the updated domain graph.

[0004] The disclosed technology also relates to systems and methods for adaptive diagnostic assessment training, including one or more server hardware computing devices or client hardware computing devices, communicatively coupled to a network, and each including at least one processor executing specific computer-executable instructions within a memory that, when executed, cause the system to: identifying, by an electronic processor, a plurality of skill nodes in a domain graph stored in an electronic memory, the plurality of skill nodes of the domain graph arranged as a directed acyclic graph; generating a first dummy node, indicating perfect proficiency; connecting the first dummy node to a root skill node of the plurality of skill nodes; generating a second dummy node, indicating perfect non-proficiency; and connecting the second dummy node to a leaf skill node of the plurality of skill nodes.

[0005] The above features and advantages of the present invention will be better understood from the following detailed description taken in conjunction with the accompanying drawings.BRIEF DESCRIPTION OF THE DRAWINGS

[0006] FIG. 1 illustrates a system level block diagram for an adaptive learning proficiency assessment system architecture.

[0007] FIG. 2 illustrates a system level block diagram for an adaptive learning proficiency system architecture, in accordance with various aspects of the techniques described in this disclosure.

[0008] FIG. 3 illustrates a system level block diagram of a content management system that facilitates a learning proficiency assessment and remediation system architecture, in accordance with various aspects of the techniques described in this disclosure.

[0009] FIG. 4 is a schematic diagram illustrating an example method and technique for an adaptive diagnostic assessment, in accordance with various aspects of the techniques described in this disclosure.

[0010] FIG. 5 is a flowchart illustrating an example method and technique for an adaptive diagnostic assessment, in accordance with various aspects of the techniques described in this disclosure.

[0011] FIG. 6. is a schematic diagram conceptually illustrating a domain graph, in accordance with various aspects of the techniques described in this disclosure.

[0012] FIG. 7. is a schematic diagram conceptually illustrating a skill node, in accordance with various aspects of the techniques described in this disclosure.

[0013] FIG. 8. is a schematic diagram conceptually illustrating a skill node determined with prior data, in accordance with various aspects of the techniques described in this disclosure.

[0014] FIG. 9A. is a schematic diagram conceptually illustrating a skill node determined without prior data, and FIG. 9B. is a schematic diagram conceptually illustrating a skill node determined with prior data, in accordance with various aspects of the techniques described in this disclosure.

[0015] FIG. 10. is a schematic diagram conceptually illustrating multiple skill nodes along with a chain of edges, in accordance with various aspects of the techniques described in this disclosure.

[0016] FIG. 11. is a schematic diagram conceptually illustrating multiple chains intersecting a skill node, in accordance with various aspects of the techniques described in this disclosure.

[0017] FIG. 12. is a schematic diagram conceptually illustrating information propagation on a domain diagram with multiple skill nodes, in accordance with various aspects of the techniques described in this disclosure.

[0018] FIGS. 13A-13C illustrate schematic diagrams conceptually illustrating different types of edges, in accordance with various aspects of the techniques described in this disclosure.

[0019] FIG. 14 is a flowchart illustrating an example method and technique for adaptive diagnostic assessment training, in accordance with various aspects of the techniques described in this disclosure.

[0020] FIG. 15 is a schematic diagram conceptually illustrating synthetic profiles, in accordance with various aspects of the techniques described in this disclosure.DETAILED DESCRIPTION

[0021] The disclosed technology will now be discussed in detail with regard to the attached drawing figures that were briefly described above. In the following description, numerous specific details are set forth illustrating the Applicant’s best mode for practicing the invention and enabling one of ordinary skill in the art to make and use the invention. It will be obvious, however, to one skilled in the art that the present invention may be practiced without many of these specific details. In other instances, well-known machines, structures, and method steps have not been described in particular detail in order to avoid unnecessarily obscuring the present invention. Unless otherwise indicated, like parts and method steps are referred to with like reference numerals.

[0022] Some computer-based learning solutions assess a learner’s proficiency for a subject solely based on a number of correct or incorrect answers for practice questions. However, such assessments can result in a large discrepancy between the learner’s measured proficiency and the learner’s actual (absolute) proficiency for a subject. For example, such a proficiency assessment may provide a binary proficiency for each obj ective of millions or trillions of obj ectives or possible learner states. This proficiency assessment may not be able to capture the uncertain and probabilistic character of the learner’s proficiency over an objective, topic, or subject. Thus, such proficiency assessments are at times incomprehensible due to the number of total objectives or learner states, inflexible, and imprecise due to the deterministic nature of binary proficiency states.

[0023] The disclosed system provides, among other things, domain graph control for adaptive learning proficiency assessment. The disclosed system may use a manageable number of skill nodes (e.g., about 1,000 skill nodes) in a domain diagram as a directed acyclic graph and may assess a proficiency of each skill node as a continuous marginal probability distribution. Thus, the disclosed system does not determine the proficiency of each objective or state as a binary proficiency but a continuous probabilistic distribution. In addition, the disclosed system may provide an efficient assessment by identifying and providing a question to a learner that has the maximum information gain of the skill nodes in a domain graph. As a result, the disclosed system may provide a minimum number of questions to the learner to obtain confident proficiencies (e.g., higher than a minimum confidence level) of objectives, topics, or subjects. Additionally, the disclosed system may generate synthetic student profiles, which correspond to true proficiencies, for skill nodes to train hyperparameters of the disclosed system using prior data or prior distributions associated with the skill nodes. Use of the synthetic student profiles can provide moreaccurate estimates of the learner’s proficiencies. Therefore, the disclosed system may assess the learner’s proficiency with a reduced or minimum number of questions in near-real time. Also, the disclosed system provides an improvement to computer-based proficiency assessment technology in that the assessments are closer to true and absolute proficiencies of a learner than previous proficiency assessment technologies.

[0024] FIG. 1 illustrates a non-limiting example of a distributed computing environment 100. In some examples, the distributed computing environment 100 may include one or more server(s) 102 (e.g., data servers, computing devices, computers, etc.), one or more client computing devices 106, and other components that may implement certain embodiments and features described herein. Other devices, such as specialized sensor devices, etc., may interact with the client computing device(s) 106 and / or the server(s) 102. The server(s) 102, client computing device(s) 106, or any other devices may be configured to implement a client-server model or any other distributed computing architecture. In an illustrative and non-limiting example, the client devices 106 may include a first client device 106A, a second client device 106B, and / or other client devices. The first client device 106A may correspond to a first learner in a class and the second client device 106B may correspond to a second learner. In some examples, the client device 106 may include one or more device controlled s) for one or more user interface input devices to receive a user input (e.g., an answer, a response, a control input, an indication, etc.) and / or user interface output devices (e.g., a display to display a question, a proficiency indication, a user profile, etc.).

[0025] In some examples, the server(s) 102, the client computing device(s) 106, and any other disclosed devices may be communicatively coupled via one or more communication network(s) 120. The communication network(s) 120 may be any type of network known in the art supporting data communications. As non-limiting examples, network 120 may be a local area network (LAN; e.g., Ethernet, Token-Ring, etc.), a wide-area network (e.g., the Internet), an infrared or wireless network, a public switched telephone networks (PSTNs), a virtual network, etc. Network 120 may use any available protocols, such as, e.g., transmission control protocol / Internet protocol (TCP / IP), systems network architecture (SNA), Internet packet exchange (IPX), Secure Sockets Layer (SSL), Transport Layer Security (TLS), Hypertext Transfer Protocol (HTTP), Secure Hypertext Transfer Protocol (HTTPS), Institute of Electrical and Electronics (IEEE) 802.11 protocol suite or other wireless protocols, and the like.

[0026] The embodiments shown in FIGS. 1 and / or 2 are respective examples of a distributed computing system and are not intended to be limiting. The subsystems and components within the server(s) 102 and the client computing device(s) 106 may be implemented in hardware, firmware, software, or combinations thereof. Various different subsystems and / or components 104 may be implemented on server 102. Learners operating the client computing device(s) 106 may initiate one or more client applications to use services provided by these subsystems and components. Various different system configurations are possible in different distributed computing environments 100 and content distribution networks. Server 102 may be configured to run one or more server software applications or services, for example, web-based or cloud-based services, to support content distribution and interaction with client computing device(s) 106. Learners operating client computing device(s) 106 may in turn utilize one or more client applications (e.g., virtual client applications) to interact with server 102 to utilize the services provided by these components. The client computing device(s) 106 may be configured to receive and execute client applications over the communication network(s) 120. Such client applications may be web browser-based applications and / or standalone software applications, such as mobile device applications. The client computing device(s) 106 may receive client applications from server 102 or from other application providers (e.g., public or private application stores).

[0027] As shown in FIG. 1, various security and integration components 108 may be used to manage communications over the communication network(s) 120 (e.g., a file-based integration scheme, a service-based integration scheme, etc.). In some examples, the security and integration components 108 may implement various security features for data transmission and storage, such as authenticating users or restricting access to unknown or unauthorized users. As non-limiting examples, the security and integration components 108 may include dedicated hardware, specialized networking components, and / or software (e.g., web servers, authentication servers, firewalls, routers, gateways, load balancers, etc.) within one or more data centers in one or more physical location(s) and / or operated by one or more entities, and / or may be operated within a cloud infrastructure. In various implementations, the security and integration components 108 may transmit data between the various devices in the distribution computing environment 100 (e.g., in a content distribution system or network). In some examples, the security and integration components 108 may use secure data transmission protocols and / or encryption (e.g., File TransferProtocol (FTP), Secure File Transfer Protocol (SFTP), and / or Pretty Good Privacy (PGP) encryption) for data transfers, etc.).

[0028] In some examples, the security and integration components 108 may implement one or more web services (e.g., cross-domain and / or cross-platform web services) within the distribution computing environment 100, and may be developed for enterprise use in accordance with various web service standards (e.g., the Web Service Interoperability (WS-I) guidelines). In an example, some web services may provide secure connections, authentication, and / or confidentiality throughout the network using technologies such as SSL, TLS, HTTP, HTTPS, WS- Security standard (providing secure SOAP messages using XML encryption), etc. In some examples, the security and integration components 108 may include specialized hardware, network appliances, and the like (e.g., hardware-accelerated SSL and HTTPS), possibly installed and configured between one or more server(s) 102 and other network components. In such examples, the security and integration components 108 may thus provide secure web services, thereby allowing any external devices to communicate directly with the specialized hardware, network appliances, etc.

[0029] A distribution computing environment 100 may further include one or more data stores 110. In some examples, the one or more data stores 110 may include, and / or reside on, one or more back-end servers 112, operating in one or more data center(s) in one or more physical locations. In such examples, the one or more data stores 110 may communicate data between one or more devices, such as those connected via the one or more communication network(s) 120. In some cases, the one or more data stores 110 may reside on a non-transitory storage medium within one or more server(s) 102. In some examples, data stores 110 and back-end servers 112 may reside in a storage-area network (SAN). In addition, access to one or more data stores 110, in some examples, may be limited and / or denied based on the processes, user credentials, and / or devices attempting to interact with the one or more data stores 110.

[0030] With reference now to FIG. 2, a block diagram of an example computing system 200 is shown. The computing system 200 (e.g., one or more computers) may correspond to any one or more of the computing devices or servers of the distribution computing environment 100, or any other computing devices described herein. In an example, the computing system 200 may represent an example of one or more server(s) 102 of the distribution computing environment 100. In another example, the computing system 200 may represent an example of theclient computing device(s) 106 of the distribution computing environment 100. In some examples, the computing system 200 may represent a combination of one or more computing devices and / or servers of the distribution computing environment 100.

[0031] In some examples, the computing system 200 may include processing circuitry 204, such as one or more processing unit(s), processor(s), etc. In some examples, the processing circuitry 204 may communicate (e.g., interface) with a number of peripheral subsystems via a bus subsystem 202. These peripheral subsystems may include, for example, a storage subsystem 210, an input / output (I / O) subsystem 226, and a communications subsystem 232.

[0032] In some examples, the processing circuitry 204 may be implemented as one or more integrated circuits (e.g., a micro-processor or microcontroller). In an example, the processing circuitry 204 may control the operation of the computing system 200. The processing circuitry 204 may include single core and / or multicore (e.g., quad core, hexa-core, octo-core, ten-core, etc.) processors and processor caches. The processing circuitry 204 may execute a variety of resident software processes embodied in program code, and may maintain multiple concurrently executing programs or processes. In some examples, the processing circuitry 204 may include one or more specialized processors, (e.g., digital signal processors (DSPs), outboard, graphics applicationspecific, and / or other processors).

[0033] In some examples, the bus subsystem 202 provides a mechanism for intended communication between the various components and subsystems of computing system 200. Although the bus subsystem 202 is shown schematically as a single bus, alternative embodiments of the bus subsystem may utilize multiple buses. In some examples, the bus subsystem 202 may include a memory bus, memory controller, peripheral bus, and / or local bus using any of a variety of bus architectures (e.g., Industry Standard Architecture (ISA), Micro Channel Architecture (MCA), Enhanced ISA (EISA), Video Electronics Standards Association (VESA), and / or Peripheral Component Interconnect (PCI) bus, possibly implemented as a Mezzanine bus manufactured to the IEEE P1386.1 standard).

[0034] In some examples, the VO subsystem 226 may include one or more device controller(s) 228 for one or more user interface input devices and / or user interface output devices, possibly integrated with the computing system 200 (e.g., integrated audio / video systems, and / or touchscreen displays), or may be separate peripheral devices which are attachable / detachable from the computing system 200. Input may include keyboard or mouse input, audio input (e.g., spokencommands), etc. As non-limiting examples, input devices may include a keyboard, pointing devices (e.g., mouse, trackball, and associated input), touchpads, touch screens, scroll wheels, click wheels, dials, buttons, switches, keypad, audio input devices, voice command recognition systems, microphones, three dimensional (3D) mice, joysticks, pointing sticks, gamepads, graphic tablets, speakers, digital cameras, digital camcorders, portable media players, webcams, image scanners, fingerprint scanners, barcode readers, 3D scanners, 3D printers, laser rangefinders, eye gaze tracking devices, medical imaging input devices, MIDI keyboards, digital musical instruments, and the like. In general, use of the term “output device” is intended to include all possible types of devices and mechanisms for outputting information from computing system 200, such as to a learner (e.g., via a display device) or any other computing system, such as a second computing system 200. In an example, output devices may include one or more display subsystems and / or display devices that visually convey text, graphics and audio / video information (e.g., cathode ray tube (CRT) displays, flat-panel devices, liquid crystal display (LCD) or light emitting diode (LED) display devices, projection devices, touch screens, etc.), and / or may include one or more non-visual display subsystems and / or non-visual display devices, such as audio output devices, etc. As non-limiting examples, output devices may include, monitors, printers, speakers, headphones, automotive navigation systems, plotters, voice output devices, modems, etc.

[0035] In some examples, the computing system 200 may include one or more storage subsystems 210, including hardware and software components used for storing data and program instructions, such as system memory 218 and computer-readable storage media 216. In some examples, the system memory 218 and / or the computer-readable storage media 216 may store and / or include program instructions that are loadable and executable on the processor(s) 204. In an example, the system memory 218 may load and / or execute an operating system 224, program data 222, server applications, application program(s) 220 (e.g., client applications), Internet browsers, mid-tier applications, etc. In some examples, the system memory 218 may further store data generated during execution of these instructions.

[0036] In some examples, the system memory 218 may be stored in volatile memory (e g., random-access memory (RAM) 212, including static random-access memory (SRAM) or dynamic random-access memory (DRAM)). In an example, the RAM 212 may contain data and / or program modules that are immediately accessible to and / or operated and executed by the processing circuitry 204. In some examples, the system memory 218 may also be stored in non-volatile storage drives 214 (e.g., read-only memory (ROM), flash memory, etc.). In an example, a basic input / output system (BIOS), containing the basic routines that help to transfer information between elements within the computing system 200 (e.g., during start-up), may typically be stored in the non-volatile storage drives 214.

[0037] In some examples, the storage subsystem 210 may include one or more tangible computer-readable storage media 216 for storing the basic programming and data constructs that provide the functionality of some embodiments. In an example, the storage subsystem 210 may include software, programs, code modules, instructions, etc., that may be executed by the processing circuitry 204, in order to provide the functionality described herein. In some examples, data generated from the executed software, programs, code, modules, or instructions may be stored within a data storage repository within the storage subsystem 210. In some examples, the storage subsystem 210 may also include a computer-readable storage media reader connected to the computer-readable storage media 216.

[0038] In some examples, the computer-readable storage media 216 may contain program code, or portions of program code. Together and, optionally, in combination with the system memory 218, the computer-readable storage media 216 may comprehensively represent remote, local, fixed, and / or removable storage devices plus storage media for temporarily and / or more permanently containing, storing, transmitting, and / or retrieving computer-readable information. In some examples, the computer-readable storage media 216 may include any appropriate media known or used in the art, including storage media and communication media, such as but not limited to, volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage and / or transmission of information. This may include tangible computer-readable storage media such as RAM, ROM, electronically erasable programmable ROM (EEPROM), flash memory or other memory technology, CD-ROM, digital versatile disk (DVD), or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or other tangible computer-readable media. This may also include nontangible computer-readable media, such as data signals, data transmissions, or any other medium which may be used to transmit the desired information and which may be accessed by the computing system 200. In an illustrative and non-limiting example, the computer-readable storage media 216 may include a hard disk drive that reads from or writes to non-removable, nonvolatile magnetic media, a magnetic disk drive that reads from or writes to a removable, nonvolatilemagnetic disk, and an optical disk drive that reads from or writes to a removable, nonvolatile optical disk such as a CD ROM, DVD, and Blu-Ray® disk, or other optical media.

[0039] In some examples, the computer-readable storage media 216 may include, but is not limited to, Zip® drives, flash memory cards, universal serial bus (USB) flash drives, secure digital (SD) cards, DVD disks, digital video tape, and the like. In some examples, the computer- readable storage media 216 may include, solid-state drives (SSD) based on non-volatile memory such as flash-memory based SSDs, enterprise flash drives, solid state ROM, and the like, SSDs based on volatile memory such as solid-state RAM, dynamic RAM, static RAM, DRAM-based SSDs, magneto-resistive RAM (MRAM) SSDs, and hybrid SSDs that use a combination of DRAM and flash memory-based SSDs. The disk drives and their associated computer-readable media may provide non-volatile storage of computer-readable instructions, data structures, program modules, and other data for the computing system 200.

[0040] In some examples, the communications subsystem 232 may provide a communication interface from the computing system 200 and external computing devices via one or more communication networks, including local area networks (LANs), wide area networks (WANs) (e.g., the Internet), and various wireless telecommunications networks. As illustrated in FIG. 2, the communications subsystem 232 may include, for example, one or more network interface controllers (NICs) 234, such as Ethernet cards, Asynchronous Transfer Mode NICs, Token Ring NICs, and the like, as well as one or more wireless communications interfaces 236, such as wireless network interface controllers (WNICs), wireless network adapters, and the like. Additionally, and / or alternatively, the communications subsystem 232 may include one or more modems (telephone, satellite, cable, ISDN), synchronous or asynchronous digital subscriber line (DSL) units, Fire Wire® interfaces, USB® interfaces, and the like. Communications subsystem 232 also may include radio frequency (RF) transceiver components for accessing wireless voice and / or data networks (e.g., using cellular telephone technology, advanced data network technology, such as 3G, 4G, 5G or EDGE (enhanced data rates for global evolution), WiFi (IEEE 802.11 family standards, or other mobile communication technologies, or any combination thereof), global positioning system (GPS) receiver components, and / or other components.

[0041] In some examples, the communications subsystem 232 may also receive input communication in the form of structured and / or unstructured data feeds, event streams, eventupdates, and the like, on behalf of one or more learners who may use or access the computing system 200. For example, the communications subsystem 232 may receive multiple skill nodes in a domain graph, questions associated with skill nodes, synthetic profdes, an assessment result, a remediation action associated with a skill node, etc. In an example, the communications subsystem 232 may be configured to receive data feeds in real-time from learners of social networks and / or other communication services, web feeds such as Rich Site Summary (RSS) feeds, and / or real-time updates from one or more third party information sources (e.g., data aggregators). Additionally, the communications subsystem 232 may be configured to receive data in the form of continuous data streams, which may include event streams of real-time events and / or event updates (e.g., sensor data applications, financial tickers, network performance measuring tools, clickstream analysis tools, automobile traffic monitoring, etc.). In some examples, the communications subsystem 232 may output such structured and / or unstructured data feeds, event streams, event updates, and the like to one or more data stores that may be in communication with one or more streaming data source computing systems (e.g., one or more data source computers, etc.) coupled to the computing system 200. The various physical components of the communications subsystem 232 may be detachable components coupled to the computing system 200 via a computer network (e.g., a communication network 120), a FireWire® bus, or the like, and / or may be physically integrated onto a motherboard of the computing system 200. In some examples, the communications subsystem 232 may be implemented in whole or in part by software.

[0042] Due to the ever-changing nature of computers and networks, the description of the computing system 200 depicted in the figure is intended only as a specific example. Many other configurations having more or fewer components than the system depicted in the figure are possible. For example, customized hardware might also be used and / or particular elements might be implemented in hardware, firmware, software, or a combination. Further, connection to other computing devices, such as network input / output devices, may be employed. Based on the disclosure and teachings provided herein, a person of ordinary skill in the art will appreciate other ways and / or methods to implement the various embodiments.

[0043] FIG. 3 illustrates a system level block diagram of a content management system that facilitates a learning proficiency assessment and remediation system 300 according to some examples. In some examples, the system 300 may include one or more database(s) 110, alsoreferred to as data stores herein. The database(s) 110 may include a learner profile 302 (e.g., a set of user data items, a learning history, scores, current proficiency indication, learner preferences, interesting objects, etc.). In such examples, the system 300 may store and / or manage the learner profile 302 in accordance with one or more of the various techniques of the disclosure.

[0044] In some examples, the database(s) 110 may include a domain graph 304 for a learner. In some examples, the domain graph 304 may include multiple skill nodes and edges. The multiple skill nodes may be arranged as a directed acyclic graph. In some examples, the domain graph 304 may represent each learner’s proficiencies in a subject (e.g., mathematics, physics, biology, literacy, a social study, etc.), a topic of the subject (e.g., algebra of mathematics, number theory of mathematics, geometry of mathematics, arithmetic of mathematics, etc.), a course (e.g., college algebra, basic physics, quantum mechanics, or any course to learn a topic for a predetermined time period), a book, and / or a job skill to perform certain tasks in a job (e.g., UX design, UI design, data analysis, photography, presentation skills, etc.). For example, when an instructor teaches a course, Algebra I, the domain graph 304 may be generated for the course. In such examples, the domain graph 304 may include multiple sets of skill nodes where each set may represent a topic in the course (e.g., basic algebra, multi-step equations, inequalities, polynomials, quadratic equations, etc.), and each set may include multiple skill nodes (e.g., expressions of one variable, expression of two variables, expression of three variables, and other concepts for the basic algebra). In further examples, the domain graph 304 may be associated with mathematics in general and include a set of skill nodes associated with Algebra I and other sets of skill nodes associated with different topics in mathematics.

[0045] A skill node in the domain graph 304 may include an estimated proficiency indication of a learning objective as a continuous marginal probability distribution. For example, the continuous marginal probability distribution may be alatent, continuous, random variable with support from 0 to 1 representing the proportion of questions that the learner is expected to answer correctly for a learning objective. In some examples, the continuous marginal distribution may include a Beta distribution representing a distribution over possible proportions of one or more questions that a learner is expected to answer correctly. The Beta distribution may be parameterized with an alpha parameter, representing a first number of correct answers at the skill node, and a beta parameter, representing a second number of incorrect answers at the skill node. Each objective’s marginal distribution may be modeled as a Beta distribution with alpha and betaparameters. Thus, the domain graph 304 as a directed acyclic graph includes all the marginals, which are Beta distributions corresponding to skill nodes. In some examples, the domain graph 304 may include a first dummy node representing perfect proficiency and a second dummy node representing perfect non-proficiency.

[0046] In some examples, the learning objective may correspond to a skill node and include a learning topic, a difficulty in a learning topic, a chapter or a portion of a course or a book. For example, the learning objective may be expressions with two variables in the Algebra I course. In such examples, one or more questions for expressions with two variables may be associated with the skill node. In further examples, the learning objective may correspond to more than one skill nodes. For example, the learning objective may be basic algebra, which include several subobjectives (e g., expressions with one variable, expressions with two variables, expressions with three variables, etc.). In such examples, each sub-objective may correspond to a skill node.

[0047] In some examples, an edge in the domain graph 304 may connect a skill node to another skill node (e.g., an adjacent skill node). The edge may include a propagation direction. The edge may include a relation type indication, which include a necessity indication, a support indication, a method indication, and / or a relation indication. In some examples, the relation type indication may represent relations of prerequisite and / or postrequisite dependency between two interconnected skill nodes via the edge. Further, multiple skill nodes using corresponding edges may be connected to a skill node or vice versa. In such examples of three skill nodes propagating to the node, the necessity indication may be like an AND-gate logic while the support indication and related indications may be weakened versions of the necessity indication. In other words, support edges are more elastic than necessity edges, and support edges are in turn more elastic than relation edges. How much the relation or support indication is more elastic than the necessity indication may be determined by a hyperparameter. For example, if the R hyperparameter is .4, a relation edge may transmit only 40% of the motion that a necessity edge would have. If the S hyperparameter is .8, a Support edge will transmit only 80% of the motion that a necessity edge would have. Basically, the motion in the chain gets dampened at more than the rate of the allnecessity case. The method indication may act like an OR-gate logic as only one prerequisite skill node may be used to succeed on a general skill node preceded by method skill nodes and edges. In some examples, the weights at which the alpha and beta values of a skill node are propagated may be determined based on the relation type of the edge. Additionally or alternative, the edgemay include a length which changes as a neighbor or a skill node connected in a chain moves in the domain graph 304 and may have a value between 0 (representing perfect non-proficiency indication) and 1 (representing perfect proficiency indication). In some examples, multiple nodes including a root skill node and a leaf skill node may be connected via a chain of edges whose lengths sum to 1.

[0048] In some examples, the database(s) 110 may include a learning course content 306. For example, the learning course content 306 may include one or more courses, one or more lessons, one or more books, one or more questions, one or more answers, or any suitable content. For examples, the one or more courses, may be associated with the domain graph. In some examples, the one or more questions may be associated with each node. Further, the one or more questions associated with a skill node may correspond the same level of difficulty. In further examples, a question may be associated with one or more skill nodes. In some examples, a question may have a correct answer and an incorrect answer, which is determined by a learner’s input of the client device 106 (e.g., a mouse click to select an answer, a keyboard input to write an answer, an audio input to record an answer, etc.). For example, a question (e.g., if x + y = 10 and x + 2y = 14, what are x and y?) may be associated with a skill node (e.g., expressions with two variables in the Algebra I course). In further examples, other algebra questions with two variables may be associated with the skill node. Further, the one or more answers may be stored in the database(s) 110 after receiving the one or more answers from the client device 106 in response to the one or more questions. An answer may include a number, a character, a word, a sentence, a text, a symbol, an indication to select one of multiple choices, or any other suitable indication where the server 102 may determine correctness and incorrectness based on the indication.

[0049] In some aspects of the disclosure, the server 102 in coordination with the database(s) 110 may configure the system components 104 for various functions. For example, the system components 104 may be configured to implement one or more of the functions described below in relation to FIGS. 4, 5, and / or 14, including, e.g., blocks 502-510 and / or 1402-1410. The system components 104 may, in some examples, be implemented by an electronic processor of the processing circuitry 204 of the server 102 of FIG. 2) executing instructions stored and retrieved from a memory of the server 102 (e.g., storage subsystem 210, computer readable storage media 216, and / or system memory 218 of FIG. 2). Accordingly, as used herein, a processor (or electronic processor), such as may execute one or more of the functions of FIG. 4, may refer to one or moreprocessors (or electronic processors), as well as other configurations as described with respect to the processing circuitry 204 of FIG. 2. For example, as used herein, an electronic processor may include a single processor, or may include multiple processors (distributed or co-located) that collectively perform a function or set of functions (e.g., where a first processor performs a first function of the set of functions, a second processor performs a second function of the set of functions, etc.).

[0050] In some examples, the system 300 may interact with the client computing device(s) 106 via one or more communication network(s) 120. In some examples, the client computing device(s) 106 may include a graphical user interface (GUI) 316 to provide the learning course content 306 including one or more questions and / or the learner profile 302 including the current proficiency indication.

[0051] FIG. 4 is a schematic diagram illustrating an example method and technique for an adaptive diagnostic assessment. The process 400 may be carried out by the server(s) 102 illustrated in FIG. 3, e.g., employing circuitry and / or software configured according to the block diagram illustrated in FIG. 2 or any other suitable apparatus.

[0052] In some examples, the process 500 may be carried out by any suitable apparatus or means for carrying out the functions or algorithm described below. For example, the server 102 may obtain a domain graph including multiple skill nodes where the skill nodes are arranged as a directed acyclic graph (e.g., at block 402). In some examples, the domain graph may be designed by a person (e.g., an expert). The server 102 may generate the domain graph by the person or receive the domain graph over the network 120 to store the domain graph in the database 110. In some examples, the server 102 may add prior data or a prior distribution to each skill node in the domain graph (e.g., at block 404).

[0053] In some examples, the server 102 may create synthetic profiles where each synthetic profile may include multiple ground truth proficiency indications corresponding to multiple skill nodes (e.g., at block 406). Based on the synthetic profiles, the server may run simulations to tune and model hyperparameters (e.g., at block 408). In some examples, the hyperparameters may include weights assigned to edges connected to the multiple skill nodes.

[0054] In some examples, based on the domain graph, the prior data, and / or the modeled hyperparameters, the server 102 may provide a diagnostic exam to the learner (e.g., at block 410).For example, the server 102 may determine a first skill node to maximize the information gain among the multiple skill nodes and provide a first question associated with the skill node. The server 102 may receive a correct or incorrect answer to the first question. In response to the correct or incorrect answer, the server 102 may update the estimated proficiency of the skill node and other estimated proficiencies of other skill nodes in the domain graph using the information propagation. Then, the server 102 or the learner may determine to stop the diagnostic exam. If the server 102 or the learner continues the diagnostic exam, the server may determine a next skill node to maximize the information gain among the multiple skill nodes and provide a next question associated with the skill node. In response to the answer to the next question, the server may further update the domain graph by updating the proficiencies of the skill node and other skill nodes. The server may repeat this process until the server 102 or the learner may determine to stop the diagnostic exam.

[0055] Then, the server 102 may provide current proficiency estimates based on the updated domain graph and / or provide a remediation action based on the diagnostic and proficiency estimates. This process is further explained in detail in connection with FIG. 5 for an adaptive diagnostic assessment and FIG. 14 for an adaptive diagnostic assessment training.

[0056] FIG. 5 is a flowchart illustrating an example process 500 and technique for an adaptive diagnostic assessment, in accordance with various aspects of the techniques described in this disclosure. The flowchart of FIG. 5 utilizes various system components that are described below with reference to FIGS. 1-4 and 6-13. In some examples, the process 500 may be carried out by the server(s) 102 illustrated in FIG. 3, e.g., employing circuitry and / or software configured according to the block diagram illustrated in FIG. 2. In some examples, the process 500 may be carried out by any suitable apparatus or means for carrying out the functions or algorithm described below. Additionally, although the blocks of the process 500 are presented in a sequential manner, in some examples, one or more of the blocks may be performed in a different order than presented, in parallel with another block, or bypassed.

[0057] At block 502, a server (e.g., one or more of the server(s) 102, also referred to as the server 102) determines a first skill node of multiple skill nodes in a domain graph stored in an electronic memory. FIG. 6 shows an example domain graph 600 including multiple skill nodes, which are arranged as a directed acyclic graph. Thus, a skill node 602 may be connected to another skill node via an edge 604 having a direction. In some examples, one skill node may be connectedone or more other skill nodes and more than one skill nodes may be connected to one skill node. The domain graph 600 may include the first node 602. In FIG. 6, the first node 602 is connected to a parent skill node and a child skill node. In other examples, the first node 602 may be a root skill node, which has no parent skill node, or a leaf skill node, which has no child node.

[0058] In some examples, the first skill node 602 may include a first estimated proficiency indication as a continuous marginal probability distribution. FIG. 7 shows an example skill node 602 including an estimated proficiency indication as a continuous marginal probability distribution. The continuous marginal probability distribution may include a latent, continuous, random variable between a perfect proficiency (e.g., 1 on x axis 702) and a perfect non-proficiency (e.g., 0 on x axis 702) such that the area under the curve sums to a unit probability. Thus, the estimated proficiency indication may represent the proportion of questions that a learner is expected to answer correctly for a learning objective as a continuous probability distribution over all possible proportions. In further examples, the estimated proficiency indication may represent a distribution over possible proportions. Accordingly, the estimated proficiency indication enables to reason systematically about all possible proportions without being committed to any single point estimate. The one or more questions may be associated with the first skill node 602. For example, when the first skill node 602 is associated with an object (e.g., expressions with two variables in Algebra I), the one or more questions associated with the first skill node 602 may include expressions with two variables (e.g., Questions 1 : x + y = 10 and 2x + y =4, Question 2: 3x - y = 10 and 2x + y = 5, etc.) to assess how much the learner may correctly answer the questions for the objective. Each objective’s marginal distribution may be modeled as a Beta distribution with alpha and beta parameters. Thus, the Beta distribution may represent distributions of expected correct answers over some large number of questions associated with the first skill node. Thus, the continuous marginal distribution may include a Beta distribution representing a distribution over possible proportions of one or more questions that a learner is expected to answer correctly. The Beta distribution may be parameterized with an alpha parameter 706, representing a number of correct answers at the first skill node 602, and a beta parameter 708, representing a number of incorrect answers at the first skill node 602.

[0059] In some examples, the first estimated proficiency indication may further include a point estimate indication of the continuous marginal probability distribution. For example, the point estimate indication of the continuous marginal probability distribution may include a mean,a mode, a median, or some other tunable point in between the mean and mode of the continuous marginal probability distribution. In other examples, the server 102 may separately calculate the point estimate indication without being included in the first estimated proficiency indication. The server 102 may report the point estimate indication as a mean, a mode, an average, a median, or some other tunable point in between the mean and mode of the continuous marginal probability distribution.

[0060] For example, when the learner answers five questions incorrectly and one question correctly associated with the first skill node (e.g., over an objective: expressions with two variables in Algebra I), the Beta distribution of the first skill node 602 may be shown as an exponentially increasing plot 710 with a first point estimate 712 (e g., a mean value of 0.166667, a mode, a mean, an average, or any suitable representing value of the distribution) and a first standard deviation value (e.g., 0.14086). When the learner answers four questions incorrectly and two questions correctly associated with the first skill node, the Beta distribution of the first skill node 602 may be shown as a graph 714 skewed to the non-proficiency (e.g., 0 on x axis 702) with a second point estimate 716 (e.g., a mean value of 0.33333, a mode, a mean, an average, or any suitable representing value of the distribution) and a second standard deviation value (e.g., 0.17817). When the learner answers three questions incorrectly and three questions correctly associated with the first skill node 602, the Beta distribution of the first skill node 602 may be shown as a normal distribution graph 718 with a third point estimate 720 (e.g., a mean value of 0.5, a mode, a mean, an average, or any suitable representing value of the distribution) and a third standard deviation value (e.g., 0.18898). When the learner answers two questions incorrectly and four questions correctly associated with the first skill node 602, the Beta distribution of the first skill node 602 may be shown as a graph 722 skewed to the proficiency (e.g., 1 on x axis 702) with a fourth point estimate 724 (e.g., a mean value of 0.66667, a mode, a mean, an average, or any suitable representing value of the distribution) and a fourth standard deviation value (e.g., 0.17817). When the learner answers one question incorrectly and five questions correctly associated with the first skill node 602, the Beta distribution of the first skill node 602 may be shown as an exponentially decreasing graph 726 with a fifth point estimate 728 (e.g., a mean value of 0.83333, a mode, a mean, an average, or any suitable representing value of the distribution) and a fifth standard deviation value (e g., 0.14086). As the learner answers questions associated with the first skill node602, the server 102 may increment the alpha parameter when the answer is correct and increment the beta parameter when the answer is incorrect.

[0061] In some examples, the server 102 may determine the first estimated proficiency indication of the first skill node based on a prior distribution of the first skill node and a likelihood function of the first skill node. For example, the first estimated proficiency indication in FIG. 7 is determined without a prior distribution but determined based on the number of correct and incorrect answer. However, FIG. 8 shows a skill node (e.g., the first skill node 602) determined with a prior distribution 802 and a likelihood function 804. For example, the first estimated proficiency indication 806 (e.g., a posterior distribution) of the first skill node 602 may be determined based on a prior distribution 802 and a likelihood function 804. In some examples, the prior distribution 802 is a Beta distribution and a likelihood function 804 of the first estimated proficiency indication 806 is binomial. In some examples, the likelihood function 804 may describe the probability of ultimately observing a certain success rate given some observed success rate on finite data. In some examples, the first estimated proficiency indication 806 may correspond to the posterior distribution as a Beta distribution, which is derived by a Beta-Binomial Bayesian model. Thus, the Beta-Binomial Bayesian model provides a way to incorporate prior distributions or beliefs associated with the first skill and a likelihood function conditioned on observed data to make more informed estimates of the first estimated proficiency indication for binary outcomes (e g., a correct or incorrect answer for a question associated with the first skill node).

[0062] For example, FIG. 9A. is a schematic diagram conceptually illustrating a skill node 900A determined without a prior distribution, and FIG. 9B. is a schematic diagram conceptually illustrating a skill node 900B determined with a prior distribution. In FIG. 9A, the probability distribution of the skill node 900A may be estimated without prior data, which is shown as a flat graph 902A while in FIG. 9B, the probability distribution is estimated based on prior data (e.g., alpha: 5 and beta 5), which is shown as a quasi-normal distribution 902B. Thus, the probability distribution of the skill node 900A in FIG. 9A may have an exponential distribution until the learner correctly answers two questions and incorrectly answers four questions associated with the skill node 900. On the other hand, the probability distribution of the skill node 900B in FIG. 9B may have a quasi-normal distribution when the learner correctly or incorrectly answers the first question associated with the skill node 900. After solving numerous questions associated with theskill node, the skill node with prior data or without prior data may have a similar proficiency distribution 904A, 904B. In some examples, the prior data may be added to each of the skill nodes in the domain graph. The prior data may include the average proficiency associated with all skill nodes based on empirical data. Thus, the prior data may communicate a point estimate associated with each skill node.

[0063] In some examples, the first skill node 602 may be connected to one or more adjacent skill nodes of the multiple skill nodes via one or more edges where the one or more adjacent skill nodes may include a second skill node that includes a second estimated proficiency and that is connected to the first skill node via a first edge of the one or more edges. Referring again to FIG. 6, the first node 602 is connected to one or more adjacent skill nodes including the second node 606 via a corresponding edge 604. In some examples, the second node 606 may be a parent node of the first node 602 or a child node of the first node 602. Each of the adjacent skill nodes may be connected to one or more its adjacent nodes. In further examples, the first node may not have a parent node or a child node. In some examples, a subset of the multiple skill nodes in the domain graph 600 may include one or more skill nodes connected from a root skill node, which has no parent skill node, to a leaf skill node, which has no child node. For example, a root skill node 612 may be connected via a first edge 622 to a second skill node 614, the second skill node 614 may be connected via a second edge 624 to a third skill node 616, the third skill node 616 may be connected via a third edge 626 to a fourth skill node 618, and the fourth skill node 618 may be connected via a fifth edge 628 to a leaf skill node 620. In some examples, any skill node among the five skill nodes 612, 614, 616, 618, 620 may be the first skill node 602 or the second skill node 606 in FIG. 6. In such examples, the root skill node 612 may not have a parent node and may be a parent node or a prerequisite of the second node 614, which may be a parent node or a prerequisite of the third node 616, which may be a parent node or a prerequisite of the fourth node 618, which may be a parent node or a prerequisite of the leaf node 620, which may not have a child node. In some aspects, those skill nodes are a chain of prerequisites, the learner’s proficiency at the root skill node to be greater than the proficiency at the second skill node, which is greater than the third skill node, etc. In other words, if the graph accurately captures a chain of true prerequisite relationships, a student’s proficiencies are expected to steadily decrease along that chain.

[0064] FIG. 10. is a schematic diagram conceptually illustrating multiple skill nodes along with a chain of edges. In some examples, the five nodes in FIG. 6 may be shown in FIG. 10. Insome examples, the multiple skill nodes of the domain graph may include a root skill node 612, which has no parent node. The root skill node 612 of the multiple skill nodes without a parent node may be connected by a root edge 1004 to a first dummy node 1002 having a perfect proficiency indication (e.g., 1 when the proficiency indication is between 0 and 1, 100% when the proficiency indication is between 0% and 100%, or any other suitable indication). In some examples, the domain graph may include multiple root skill nodes, which are connected to the first dummy node 1002. In some examples, the multiple skill nodes of the domain graph 600 may include the leaf skill node 620, which has no child node. The leaf skill node 620 of the multiple skill nodes without a child node may be connected by a leaf edge 1008 to a second dummy node 1006 having a perfect non-proficiency indication (e.g., 0 when the proficiency indication is between 0 and 1, 0% when the proficiency indication is between 0% and 100%, or any other suitable indication). In some examples, the domain graph may include multiple leaf skill nodes, which are connected to the second dummy node 1006. In some examples, a path connecting the first dummy node 1002 to the second dummy node 1006 may include a chain of edges 1004, 622, 624, 626, 628, 1008 where lengths corresponding to the edges of the chain may sum to the perfect proficiency indication. In some examples, the chain of edges may include the first edge connecting to the first skill node 602 in FIG. 6 or any skill node in FIG. 10. In some examples, the edges in FIG. 10 are two dimensional. Visually, true lengths of the edges may be represented geometrically by projecting the nodes’ point estimate locations down onto a one-dimensional line (effectively taking only the X component of a two-dimensional vector). Since this nuance is an artifact of drawing the graph in two dimensions, however, and does not impact computation on the graph as stored in a computer’s memory (which is not geometric). That is, the edge lengths in FIG. 10 may not be to scale insofar as they are two- dimensional vectors, not one-dimensional vectors as they become when projected onto the X axis of FIG. 10

[0065] In some examples, the chain may include first edges connecting to the first skill node (e g., the first skill node 602 in FIG. 5 or any skill node in FIG. 10) to the first dummy node 1002 and second edges connecting to the first skill node to the second dummy node 1006. The sum of the first lengths corresponding to the first edges and the second lengths corresponding to the second edges may be the perfect proficiency indication. In some examples, the point estimate associated with the first skill node may include a probability being the perfect proficiency indication subtracting a sum of first lengths of first edges in the chain connecting the first skillnode to the first dummy node. Tn further examples, the point estimate associated with the first skill node may include a probability being a sum of second lengths of second edges in the chain connecting the first skill node to the second dummy node. Thus, the multiple nodes in the domain graph may have decreasing proficiencies from the perfect proficiency indication (the left in FIG. 10 or the top in FIG. 6) to the perfect non-proficiency indication (the right in FIG. 10 or the bottom in FIG. 6).

[0066] In further examples, a skill node in the domain graph may have multiple parent skill nodes and / or multiple child skill nodes. For example, FIG. 11 is a schematic diagram 1100 conceptually illustrating two separate chains intersecting a skill node 614. The diagram 1100 includes the chain of edges 1004, 622, 624, 626, 628, 1008 and skill nodes (e.g., the first dummy node 1002, the root skill node 612, the second skill node 614, the third skill node 616, the fourth skill node 618, the leaf node 620, and the second dummy node 1006) shown in FIG. 10. The diagram 1100 shows another chain of edges and skill nodes intersecting the second skill node 614. The different chain may also include five skill nodes (skill node A 1102, skill node B 1104, skill node C or second skill node 614, skill node D 1106, and skill node E 1108) connected to the first dummy node 1002 and the second dummy node 1006. In the chain, the skill node A 1102 may be another root skill node connected to the first dummy skill node 1002 while the skill node E 1108 may be another leaf skill node connected to the second dummy skill node 1006. Thus, in the diagram 1100, the second skill node 614 may be a child skill node or a postrequisite not only to the root node 612 but also to the skill node B 1104. Similarly, the second skill node 614 may be a parent skill node or a prerequisite to both the third skill node 616 and the skill node C 1106. So the second skill node 614 may be implicated in at least two different chains of parent skill nodes or prerequisites.

[0067] In some examples, to determine the first skill node, the server 102 may use information gains of all skill nodes in the domain graph. The information gain may be calculated using a weighted combination of two values: the expected information gain if the learner answers the question for that node correctly and the expected information gain if the learner answers the question for that node incorrectly. To find a question that will maximize the expected information gain, the calculations shown in the equation shown below may be performed for each skill node in the domain graph. Then, a question associated with the skill node with the maximum expected information gain may be served to student, maximally improving thereby our proficiency estimates.

[0068] In some examples, to determine the first skill node, the server 102 may, for each skill node of the multiple skill nodes of the domain graph corresponding to the multiple questions, determine multiple gains between multiple current point estimates corresponding to multiple other skill nodes than a respective skill node in the multiple skill nodes and multiple hypothetical point estimates corresponding to the multiple other skill nodes if an answer to a respective question associated with the respective skill node is correct and incorrect, and determine potential information gain for the respective skill node based on the plurality of gains, and an estimated proficiency indication of the respective skill node. In some examples, a current point estimate may indicate a mean, a mode, an average, a median, or some other tunable point in between the mean and mode of the continuous marginal probability distribution of a skill node while a hypothetical point estimate is changed based on a propagating effect if an answer to a question associated with a served skill node is correct or incorrect. In such examples, to determine the first skill node, the server 102 may determine the first skill node corresponding to a maximum gain based on the potential information gain for each skill node of the plurality of skill nodes. In some examples, to determine the first skill node, the server 102 may, for each skill node of the multiple skill nodes corresponding to the multiple questions, determine multiple first gains of the multiple gains between multiple first current point estimates corresponding to the multiple other skill nodes and multiple first hypothetical point estimates corresponding to the multiple other skill nodes if the answer to the respective question associated with the respective skill node is correct. In such examples, to determine the first skill node, the server 102 may further, for each skill node of the multiple skill nodes corresponding to the multiple questions, determine multiple second gains of the plurality of gains between a plurality of second current point estimates corresponding to the plurality of other skill nodes and a plurality of second hypothetical point estimates corresponding to the plurality of other skill nodes if the answer to the respective question associated with the respective skill node is incorrect. In such examples, to determine the potential gain for each skill node, the server 102 may, for each skill node of the plurality of skill nodes corresponding to the plurality of questions, determine the potential information gain for the respective skill node based on: the estimated point proficiency indication associated with the respective node multiplied by a first sum of the plurality of first gains, and an estimated point non-proficiency indication associated with the respective node multiplied by a second sum of the plurality of second gains.

[0069] In some examples, to determine the first skill node, the server 102 may receive one or more selected objectives corresponding to a subset of the multiple skill nodes, identify a subset of current point estimates corresponding to the subset of the multiple skill nodes, determine a subset of hypothetical point estimates corresponding to the subset of the multiple skill nodes, and determine the first skill node based on the subset of hypothetical point estimates. In some examples, the served questions need not come from within the subset used to calculate information gain. Rather, the served questions can come from any related skill node. In such examples, to determine the first skill node, the server 102 may further determine a potential information gain for a respective skill node of the plurality of skill nodes for each skill node of the plurality of skill nodes corresponding to the plurality of questions based on a point estimate indication of the respective skill node and one or more gains corresponding to the subset of the plurality of skill nodes between the subset of current point estimates and the subset of hypothetical point estimates. In some examples, to determine the first skill node based on the subset of hypothetical point estimates, the server 102 may determine the first skill node corresponding to a maximum gain based on the potential information gain for each skill node of the plurality of skill nodes.

[0070] In some examples, when calculating the expected information gain associated with a prospective served skill, one weights both the probably of a correct answer to that served skill (p(correct on served)) and the probability of an incorrect answer to that served skill (l-p(correct on served)) such that the two weights sum to one and reflect a current point estimate for that skill:E(info gain) = n=# nodes p(correct on served) ( current point estimate — updated point estimate)2k=l n=# nodes+ (1 — p(correct on served)) ( current point estimate — updated point estimate)2fc=iThe # of nodes may be all skill nodes in the domain graph or a subset of the multiple nodes in the domain graph corresponding to selected objectives (e.g., by an instructor). In some examples, the point estimate used to calculate the p(correct on served) weight may differ from the point estimate used to calculate the changes within the summations. For example, for purposes of weighting the two possible outcomes (a correct versus an incorrect answer), one may prefer to minimize the impact of outliers and adopt the marginal distribution’s mode as a point estimate, whilenevertheless employing the distribution’s mean (more sensitive to outliers) as a point estimate within the summations.

[0071] The rationale for squaring the changes to each proficiency is that larger corrections to the current estimates may be viewed as exponentially more valuable. Just as errors are often squared when evaluating a model’s accuracy, the updated proficiencies may be interpreted as corrections to the model’s currently erroneous point proficiency estimates.

[0072] In addition, for purposes of performing these millions of calculations as efficiently as possible, the propagated alpha and beta increments may be precalculated for every possible served node and every possible outcome (correct and incorrect outcomes). This precomputation may create one large matrix of propagated alpha increments (for correct answers), and one large matrix of propagated beta increments (for incorrect answers), which enable the system to perform millions of calculations in milliseconds. The rows of the alpha increments matrix may represent the equivalent increases to all skill nodes if a correct answer is observed for the skill node corresponding to a given row. The rows of the beta increments matrix may represent the equivalent increases for all skill nodes if an incorrect answer is observed for the skill node corresponding to a given row. In some examples, these matrices may be used to efficiently calculate (via elementi- wise matrix additions) what the updated proficiencies would be for every possible outcome. For every node and for every outcome (if correct and if incorrect for every possible node), the expected information gains associated with service of that node may be precalculated in order to minimize the computations used to identify optimal items at run time.

[0073] The tunable formula to calculate the probability of getting a correct answer on a given node, in turn, may be based on the alpha and beta parameters of the Beta distribution, a distribution supported by all possible proficiency rates between 0 and 1. For purposes of reducing a Beta distribution to the single point estimate used to calculate information gain, either the mean of the distribution (if the tunable parameter x is set to 0), the mode of the distribution (if the tunable parameter x is set to 1), or some compromise value in between those two extremes (0 < x < 1) which might be something closer to the median value may be used:,1alpha - x p(correct on served) = — — - - alpha+beta - 2xThis formula is based on the formula for calculating the mode of a Beta distribution (where x=l). However, when alpha and beta are small enough, this formula will occasionally yield probabilitiesthat fall outside the permissible range of 0 to 1 . In such edge cases, values less than 0 are truncated to 0, and values greater than 1 are truncated to 1 — in keeping with the mathematical properties of the mode of a two-parameter Beta distribution.

[0074] In some examples, the server 102 may determine the first skill node based on one or more objectives an instructor selects. For example, the learner may take Algebra I course from an instructor. In the course, the instructor may select multiple objectives 630 shown in FIG. 6 in the Algebra I course to test the learner in an exam. In such examples, an objective may include the expressions with two variables, which is associated with the first skill node. In some examples, an objective 630 the instructor selects may be mapped to a skill node 606, which has multiple child skill nodes associated with specific objectives. In other examples, an objective the instructor selects may be general such that the objective may be associated with multiple skill nodes. In further examples, the server 102 may select the first skill node based on a level of difficulty. For example, the learner may take Algebra I course from an instructor. During a test, the server 102 may select the first skill node because the first skill node is associated with a basic algebra. In further examples, the server 102 may select the first skill node based on learner’s profile. For example, when the learner’s profile indicates that the learner is good at the expressions with two variables, the server 102 may select the first skill node

[0075] Returning to FIG. 5, at block 504, the server 102 provides a first question of the one or more questions to a learner. For example, the server 102 may transmit (e.g., via the communication network 120) the first question associated with the first skill node to cause the first question to be displayed on a graphical user interface (GUI) or produced on a speaker of the client device 106 for the learner. In some examples, the server 102 may provide the first question to the learner by transmitting the first question and causing the client device 106 to output the first question as a written sentence displayed on a display of the client device 106 or as a spoken sentence output via a speaker of the client device 106.

[0076] At block 506, the server 102 receives a first answer from the learner for the first question. For example, the server 102 may receive the first answer in response to the first question where the learner inputs the first answer (e.g., using an input device) to the client device 106. Then, the client device 106 may send the first answer to the server 102 (e.g., via the communication network 120). The first answer may include a number, a character, a word, a sentence, a text, asymbol, an indication to select one of multiple choices, or any other suitable indication where the server 102 may determine correctness and incorrectness based on the indication.

[0077] At block 508, the server 102 updates the domain graph based on the first answer. For example, the server 102 updates the domain graph by: updating the first estimated proficiency indication of the first skill node and updating the second estimated proficiency indication of the second skill node. Thus, in some examples, a marginal probability distribution for each skill node may be maintained or calculated instead of maintaining or directly calculating a joint probability distribution across the domain graph.

[0078] In some examples, to update the first skill node, the server 102 may update the alpha parameter of the Beta distribution associated with the first skill node in response to the first answer being correct by adding a constant hyperparameter to the alpha parameter (e.g., incrementing the first number of correct answers of the alpha parameter). Also, to update the first skill node, the server 102 may update the beta parameter of the Beta distribution associated with the first skill node in response to the first answer being incorrect by adding the constant hyperparameter to the beta parameter (e.g., incrementing the second number of incorrect answers of the beta parameter). Updating the alpha parameter upon the correct answer or the beta parameter upon the incorrect answer is described above in connection with FIGS. 7, 8, 9A, and 9B with and without prior data. For example, when the alpha and beta parameters are 1 and the learner incorrectly answers the question associated with the first skill node, the number of incorrect answers of the beta parameter is incremented and the point estimate of the estimated probability indication moves toward perfect non-proficiency (e.g., from 0.5 to 0.33 in FIG. 9A and from 0.5 to 0.45 in FIG. 9B).

[0079] In some examples, as part of updating the first skill node, the server 102 may change the location of the first skill node’s point estimate within the domain graph. In the examples of FIG. 10, the five skill nodes may have decreasing proficiencies from perfect proficiency (e.g., the leftmost side of the diagram 1000) to non-perfect proficiency (e.g., the rightmost side of the diagram 1000). When the learner correctly answers a question associated with the second node 614 in FIG. 10, the estimated proficiency indication of the second node 614 may be improved or increase and move toward the first dummy node 1002, representing a perfect proficiency indication. In such examples, the lengths of one or more skill edges placed to the left (or the perfect proficiency indication side of the second node 614 in FIG. 10) may be linearly shortened while lengths of oneor more skill nodes placed to the right (the perfect non-proficiency indication side of the second node 614) may be linearly lengthened to ensure that the updated complete chain of edges continues to sum to 1.0 or perfect proficiency. In other examples, when the learner incorrectly answers a question associated with the second node 614 in FIG. 10, the estimated proficiency indication of the second node 614 may decrease and move toward the second dummy node 1006 or the perfect non-proficiency indication. In such examples, lengths of one or more skill edges placed to the left (the perfect proficiency indication side of the second node 614 in FIG. 10) may be linearly lengthened while the lengths of one or more skill edges placed to the right (the perfect non- proficiency indication side of the second node 614) may be linearly shortened to ensure that the updated chain of edges continues to sum to 1.0 or perfect proficiency.

[0080] In the examples of FIG. 11, when the learner correctly answers a question associated with the second node 614 in FIG. 11, the estimated point proficiency indication of the second node 614 may be improved or increase and move toward the first dummy node 1002, representing the perfect proficiency indication. In such examples, the lengths of one or more skill edges (e.g., skill node 612 in the first chain of edges 1004, 622, 624, 626, 628, 1008 and skill nodes 1102, 1104 the second chain of edges 1112, 1114, 1116, 1118, 1120, 1122) placed to the left (the perfect proficiency indication side of the second node 614 in FIG. 11) may be linearly shortened while the lengths of one or more skill edges (e g., skill nodes 616, 618, 620 in the first chain of edges and skill nodes 1106, 1108 the second chain of edges) placed to the right (the perfect non- proficiency indication side of the second node 614) may be linearly lengthened to ensure that the updated chain of edges continues to sum to 1.0 or perfect proficiency. In other examples, when the learner incorrectly answers a question associated with the second node 614 in FIG. 11, the estimated point proficiency indication of the second node 614 may decrease and move toward the second dummy node 1006, representing the perfect non-proficiency indication. In such examples, lengths of one or more skill edges (e.g., skill node 612 in the first chain of edges and skill nodes 1102, 1104 the second chain of edges) placed to the left (the perfect proficiency indication side of the second node 614 in FIG. 10) may be linearly lengthened while lengths of one or more skill edges (e.g., skill nodes 616, 618, 620 in the first chain of edges and skill nodes 1106, 1108 the second chain of edges) placed to the right (the perfect non-proficiency indication side of the second node 614) may be linearly shortened to ensure that the updated chain of edges continues to sum to 1.0 or perfect proficiency.

[0081] In some examples, when the learner correctly or incorrectly answers to a question associated with a skill node, the estimated proficiency indications of other skill nodes in addition to that of the skill node may be affected based on propagation. FIG. 12. is a schematic diagram conceptually illustrating information propagation on a domain diagram with multiple skill nodes. When the first answer to the first question associated with a skill node 1202 (e.g., the first skill node 602 or any skill node) in the domain diagram 600, the server 102 updates the estimated proficiency indication of the skill node 1202 by updating the alpha parameter or beta parameter of the Beta distribution of the skill node 1202. In addition to the updated estimated proficiency indication of the skill node 1202, the server 102 may update one or more first adjacent skill nodes 1204 to the skill node 1202. The one or more first adjacent skill nodes 1204 may be one-step away from the skill node 1202. Each skill node of the one or more first adjacent skill nodes 1204 may have one or more second adjacent skill nodes 1206. The server 102 may update one or more second adjacent skill nodes 1204 based on the one or more first updated adjacent skill nodes 1202. The one or more first adjacent skill nodes 1204 may be two steps away from the skill node 1202, depending on the directions of the arrows associated with both the first and second steps. In accordance with D-separation theory, stepping backward twice in row (up one edge arrow to a parent node and then up a second edge arrow to that node’s parent), stepping forward twice in a row (down one edge arrow to a child node and then down a second edge arrow to that node’ s child), or stepping backward then forward (up one edge arrow to a parent node and then down another edge arrow to a different child node) are all permissible combinations of steps for propagation. By contrast, information may not propagate by stepping forward down one edge arrow (to a child node) and then immediately stepping backward up another edge arrow (to a different parent node) unless some other, permissible path connects those same two nodes and thereby permits information to propagate between them via that alternative, permissible path. In the event that two different, permissible paths between two nodes would produce two differing propagation results, some examples may resolve such conflicts by adopting the larger of the two propagated results. The server 102 may change the estimated proficiency indications of the one or more second adjacent skill nodes 1206 less than the changes to the estimated proficiency indications of the one or more first adjacent skill nodes 1204. In this manner, the server 102 may update most or even, depending on the structure of the graph’s edges, all skill nodes directly or indirectly connected to skill node 1202. Thus, the change in the estimated proficiency indication of the skill node 1202based on the correct or incorrect answer to the question associated with the skill node 1202 may have a propagating effect on other skill nodes in the domain graph 600.

[0082] In some examples, when the skill node 1202 may be determined to maximize the information gain as described in connection with block 502, the information propagation to other skill nodes upon the correct and incorrect answer to the questions associated with the skill node 1202 may be precomputed prior to run time. In some examples, the system precomputes the increment to either the alpha or the beta parameter if the alpha or the beta of a given node is increased by one. Thus, these increments may always be relative and never be absolute values. When the learner answers the question associated with skill node 1202, the hypothetical effect or non-effect of that answer on every other node may be precomputed in advance and stored in a pair of vectors. One vector may capture the effect (or non-effect) that a correct answer on node 1202 will have on the alpha parameter of every node on the entire domain graph. A second vector may capture the effect (or non-effect) that an incorrect answer on node 1202 will have on the beta parameter of every node on the entire domain graph. In some examples, the vectors associated with all of the nodes may then be collected into square matrixes for optimally efficient calculations. Since these matrixes capture the hypothetical changes to the parameters of the Beta distributions associated with all nodes of the domain graph for all possible served questions and all possible outcomes (correct and incorrect answers), element-wise operations using these matrixes affords an extremely efficient means of calculating the prospective information gains associated with all possible scenarios and outcomes. By vectorizing the alpha and beta parameters associated with a given student’s current marginal proficiency estimates, one can very efficiently calculate what the student’s updated alpha and beta parameters would become under all possible served nodes and all possible outcomes: one simply adds the precalculated vectors of prospective increments to the existing vectors of current parameter values to determine what each prospective update would yield. By collecting all of these vectors into matrices, one may then very efficiently compute the hypothetical, prospective point estimates that would be produced under all possible future scenarios. In other examples, even though the stateless matrices of alpha and beta increments continue to be used for purposes of the large-scale information gain calculations in 502, where computational efficiency remains paramount, the server 102 may nevertheless use more customized, personalized, computationally expensive, stateful increments when performing the actual updates to a student’s profile in 508 after receiving a particular outcome to a particularquestion served in 506. Because the prospective information gain calculations in 502 may use attention to all possible skill-serving scenarios and all possible outcomes from each of those scenarios, computational efficiency remains more desirable for 502 than it does for 508. Accordingly, even if one chooses to use stateless, increment-based precalculations for purposes of 502, one may nevertheless prefer to use more exact but more computationally demanding stateful methods, based directly on stateful current point estimates instead of indirectly on matrices of parameter increments, for purposes of the updates in 508.

[0083] In some examples, the propagation may use information from priors (e.g., to populate the precomputed increment matrices in 502) and from the learner’s current (estimated) state in 508. The propagation may be based on the alpha and beta parameterization of the Beta distribution. In further examples, when direct access to the learner’s current point estimates is used, the propagation may use an alternative parameterization of the Beta distribution in terms of a mean parameter and a variance or precision parameter. In some examples, the propagation logic may populate the chains of edges with empty nodes, which are the first and the second dummy nodes. Also, the propagation logic may operate directly on existing, stateful point estimates or mean values instead of operating indirectly on them via stateless matrices of parameter increments. That alternative propagation logic is explained below based on the examples of FIG. 11.

[0084] Referring again to FIG. 11, two node chains intersect at the second skill node 614. In some examples, either a correct or incorrect answer may be observed at the second skill node 614, and all of the edges on the graph may be treated as full necessity edges. Both the pre-update and post-update point estimates associated with the second node 614 are known. Thus, exactly how much that estimate has moved as a result of the new information can be calculated by simple subtraction. The pre-update point estimates associated with the other nodes on these chains are also known. Employing the logic of maximum entropy, one may desire to incorporate the newly acquired information about node 614 into all the other components of the node chains in a manner that minimally disrupts the existing estimates associated with the other nodes in each chain.

[0085] To achieve that end, one may, because each chain of edges sums to one, interpret each of the chains as a single Dirichlet distribution. Such an interpretation is especially attractive because the marginal distributions of a Dirichlet distribution are necessarily all Beta distributions, as are the marginal distributions associated with every skill node in our domain graph. Moreover, because Dirichlet distributions exhibit unique neutrality and aggregation principles, the shape ofthe Beta distribution associated with node 614 remains mathematically independent of the number of nodes that happen to lie above or below it on any given chain. In other words, if one only cares about node 614’s distribution, one can effectively ignore nodes 612, 1106, 1108, 1102, 1104, 616, 618, 620, 1002, and 1006. The node 614’s distribution does not depend on the locations of the nodes lying above or below it on the chain (nodes 612, 1106, 1108, 1102, 1104, 616, 618, 620, 1002, and 1006), or on the lengths of the individual edges connecting those nodes (edges 1004, 1118, 1120, 1122, 1112, 1114, 1116, 1118, 622, 624, 626, 628, and 1008). To determine the shape of the Beta distribution associated with node 614 — provided one strictly enforces the condition that the edges of every chain connecting the first dummy node to the second dummy node sum to one — all one needs to know is the sum of the lengths of the edges that connect node 614 to either of the two perfect dummy nodes, either node 1002 or node 1006. And that sum will remain the same regard of which path one travels because that sum depends solely on the distance between node 614 and the dummy node, a node is common to all the chains on the graph. This independence of node 614 from the particulars of any of the chains to which it belongs is enormously valuable because it enables one to reason independently about the effects of a change in node 614 upon all of the subchains to which node 614 belongs, which enables one, in turn, to preserve as much of the existing information as possible within each of those separate subchains.

[0086] Specifically, by interpreting each node chain as a Dirichlet distribution and by exploiting the neutrality and aggregation properties of the Dirichlet distribution, the disruption to the other nodes may be minimized by preserving the relative relationships among the proficiency estimates included within each of the four subchains 1132, 1134, 1136, 1138 connected to the second node 614. The first subchain may include edges 1004, 622; the second subchain may include edges 624, 626, 628, 1008; the third subchain may include 1112, 1114, 1116; and the fourth subchain may include edges 1118, 1120, 1122. If the edges within the first, second, third, and fourth subchains 1132, 1134, 1136, 11338 remain in exactly the same proportions to each other as they were prior to the update, as much information as possible may be preserved within each of the four subchains. Because there is no reason to believe that the new information acquired about node 614 has altered the relationships within each subchain to which node 614 is connected, those subchain relationships may be maximally retained by linearly rescaling each subchain. Such rescaling accords with the Dirichlet’s unique neutrality property and ensures that each completeedge chain (from dummy node 1002 to dummy node 1006) continues to sum to one after all the skill nodes along the subchains have been updated.

[0087] If the second node 614 was answered correctly, its proficiency estimate will increase. It will shift leftward, which will shift all the other proficiencies in the first and third subchains 1132, 1136 leftward as well. The first and third subchains 1132, 1136 will both shrink linearly, while the second and fourth subchains 1134, 1138 will be both stretch linearly. Conversely, if the second node 614 was answered incorrectly, it will move rightward because its proficiency estimate will decrease (how much it moves depends on how much data backed up the estimate — confident estimates move less than uncertain estimates). In that scenario, the third and third subchains 1132, 1136 both get stretched out, and the second and fourth subchains 1124, 1148 will both shrink. As long as all the edges in a subchain are all necessity edges, the relative proportions of those edges will look the exact same post-update as they did pre-update. All of the nodes will have moved, and the overall shape of the graph will have shifted. But the relative shapes / proportions within each subchain will remain unchanged due to the neutrality properties of the Dirichlet. Note that by interpreting each chain as a Dirichlet, it may be implicitly assumed that the same amount of data may exist about every node on the graph. While this assumption will never strictly hold true in practice, it nevertheless remains a useful fiction to maintain for purposes of propagation and can, in addition, be justified by the fact that selecting served nodes based on maximizing expect information gain helps to preserve a relatively uniform distribution of data across the operative subgraphs. Moreover, even if the edge lengths and node locations have been updated based on the fiction of assumed uniform distribution of data across the graph, such that the sum of the alpha and beta parameters for each node remain constant across the graph, by transforming each propagated shift in a point estimates back into an single-parameter increment, either to an alpha parameter (in the case of a correct answer) or to a beta parameter (in the case of an incorrect answer), one may abandon the fiction of a uniform data distribution when performing the actual parameter updates. Even though one may have fictionally assumed a uniform data distribution when calculating the desired linear rescalings of all the point estimates in all the affected subchains, by translating the desired movement for each point estimate back into a singleparameter incremental update (to either the alpha parameter or the beta parameter), the resulting update to each affected point estimate will continue to reflect the actual amount of data associated with each node. In other words, the point estimate for a node about which one has collected lessdata will be more powerfully affected by a given increment than would a node about which one has collected less data. In that sense, reliance on a fictional of uniformly distributed data does not prevent the resulting incremental, single-parameter updates from continuing to vary based on the actual amount of data collected, as reflected in the actual values of the alpha and beta parameters prior to the update. Even though the calculation of the appropriate single-parameter increment value may have been based on the fiction of uniformly distributed data, that fiction never directly uses that the sums of the alpha and beta parameter values remain constant across the domain graph. The sum of the alpha and beta parameters for any given node remains free to vary from skill node to skill node, based on the differential amounts of data informing each skill node’s estimate.

[0088] In further examples, when a subchain combines consists of mixtures of different edge types, a different rule may be applicable because support edges are more elastic than necessity edges, and support edges are in turn more elastic than relation edges. If the R hyperparameter is .4, a relation edge will transmit only 40% of the motion that a necessity edge would have. And if the S hyperparameter is .8, a Support edge will transmit only 80% of the motion that a necessity edge would have. Thus, non-necessity edges can dampen information propagation more quickly than do necessity edges. An example will illustrate this more concretely.

[0089] In some examples, when the answer on the second skill node 614 was incorrect, its estimated proficiency indication or point estimate moved from .75 to .5 as a result. Below is a table capturing the resulting effects of that movement (at the red node) with different configurations of edges (again assuming the fiction that all node estimates are based on identical confidence and data):

[0090] In this case, the relative lengths of the edges within the first subchain 1132 remain unchanged, and the relative lengths of the edges within the fourth subchain 1138 remain unchangedas well because the necessity edges are all just linearly rescaled to accommodate the second skill node’s movement. In this example, the first subchain 1132 expands to double its former size, while the fourth subchain 1138 shrinks to 2 / 3rds of its previous size. Again, within the first subchain 1132, and within the fourth subchain 1138, the relative relationships are all preserved by linearly rescaling each subchain.

[0091] In further examples, if the edge 628 connecting the fourth and leaf nodes 618, 620 is changed to a support edge, which has only an 80% chance of being operative, then there is slightly less movement in the leaf skill node 620 as a result of the uncertainty as to whether the movement in the fourth node 618 did or did not affect that final link in the chain:

[0092] In further examples, if the third edge 624 is merely a support edge, the uncertainty associated with it will result in less movement in all subsequent nodes on that subchain 1134. All subsequent edges and nodes will be impacted by the dampening effect “upstream” from them. For example, the edge 624 between the second and third nodes 614, 616 will shrink and become the same size as the fourth and fifth edges 626, 628 following the update. That relative shrinkage means that the fourth and leaf nodes 618, 620 move slightly less than they would have had the second node 614 been 100% necessary for success on its subsequent postrequisites:

[0093] As this example illustrates, the propagation logic (like the profile generation technique) may take a holistic view of the subchains, which in turn use customized, sequential updates that account for both the edge types and the learner’s current state.

[0094] FIGS. 13A-13C illustrate schematic diagrams conceptually illustrating different types of edges, in accordance with various aspects of the techniques described in this disclosure.As explained above, four types of edges (i.e., necessity, support, method, and relation) for information propagation exist. In diagram 1302 of FIG. 13A, the four types of edges and four different hyperparameters in each type of edges may exist when information is propagated from one skill node to one skill node. The hyperparameter W may represent the proportional amount by which an increase to the alpha parameter of a child node will increase the alpha parameter of its parent node if the two nodes are connected by a necessity edge. The hyperparameter X may represent the proportional amount by which an increase to the beta parameter of a parent node will increase the beta parameter of its child node if the two nodes are connected by a necessity edge. The hyperparameter Y may represent the proportional amount by which an increase to the alpha parameter of a parent node will increase the alpha parameter of its child node if the two nodes are connected by a necessity edge. The hyperparameter Z may represent the proportional amount by which an increase to the beta parameter of a child node will increase the beta parameter of its parent node if the two nodes are connected by a necessity node. The hyperparameter S may represent the proportional amount by which the W, X, Y and Z parameters are each shrunk if the parent and child nodes are connected by support type edge instead of a necessity type. If the connecting edge is a method type, no such scalar is used. The hyperparameter R may represent the proportional amount by which the W, X, Y and Z parameters are each shrunk if the parent and child nodes are connected by a relation type edge instead of a necessity type. In diagram 1304 of FIG. 13B, the four types of edges and four different hyperparameters in each type of edges may exist when information is propagated from one skill node to multiple skill nodes. If one child node’ s alpha parameter has been incremented by value A and another child node’ s alpha parameter has been incremented by value B and both children are connected by necessity edges to a common parent node, then that parent may be incremented by the maximum of the hyperparameter W multiplied by the value A and the hyperparameter W multiplied by the value B. If a parent node’s beta parameter has been incremented by some value then the hyperparameter X determines the proportion of that value that gets propagated to any child node connected by a necessity edge. If a parent node’s alpha parameter has been incremented by some value then the hyperparameter Y determines the proportion of that value that gets propagated to any child node connected by a necessity edge. If one child node’s beta parameter has been incremented by value A and another child node’s alpha parameter has been incremented by value B and both children are connected by necessity edges to a common parent node, then that parent may be incremented by the maximumof the hyperparameter W multiplied by the value A and the hyperparameter W multiplied by the value B. If one of the relevant edges is a support type instead of a necessity type, then its propagated value shrinks by S times W, X, Y, or Z. By definition, a child node connected to a parent node by a method edge should always have more than one parent, so the configurations in the final column of 1304 should never arise.

[0095] In diagram 1306 of FIG. 13C, the four types of edges and four different hyperparameters in each type of edges may exist when information is propagated from many skill nodes to one skill node. If the alpha parameter of a child node is increased by A, then the alpha parameter of each parent node connected by a necessity type edge should increase by the hyperparameter W multiple by A. If the beta parameters of two parent nodes are increased by A and B, respectively, with a third parent whose value remains unknown, then the beta parameter of a child connected to them by necessity type edges will increase by 1-((1-A)(1-B)(.5)). If the alpha parameter of a child node is increased by A, then the alpha parameter of each parent node connected by a necessity type edge should increase by the hyperparameter W multiple by A. If the alpha parameters of two parent nodes are increased by A and B, respectively, with a third parent whose value remains unknown, then the alpha parameter of a child connected to them by necessity type edges will increase by A times B times .5. If a child node’s beta parameter increases by A, then the alpha parameter of each parent node will proportionally increase by A*Z*2A(number of parents - l) / (2Anumber of parents -1). If one of those parents is connected by a support type edge instead of a necessity type edge, then the value propagated through that edge must be shrunk by the hyperparameter S when plugged into those formulas. If the parents are connected by method edges, then an increment to the alpha parameter of a child will result in a proportional increase of W*2A(number of parents- l) / 2A(number of parents- 1) to each of the parents. If the beta parameter of one parent connected by a method edge is increased, then the proportional increase to the beta parameter of its parent node will be X / 2A(number of parents-1). If the alpha parameter of a parent node is increased, then the proportional increase to its child node will be determined by the hyperparameter Y. If the beta parameter of a child node is increased, then the proportional increase to its parent node will be determined by the hyperparameter Z. In some examples, a hyperparameter in each type of edges may indicate how much information will be transferred to the adjacent node and, indirectly, how far the adjacent node’s point estimate willmove in response to the change of the estimated proficiency indication or the point estimate of the respective skill node.

[0096] In some examples, to update the second estimated proficiency indication of the second skill node, the server 102 may update the lengths of the edges of the chain of edges to be the perfect proficiency indication after updating the second estimated proficiency indication of the second skill node. In some examples, the second estimated proficiency indication of the second skill node is updated based on at least one of: a correctness of the first answer, a relation type indication, a propagation direction, or a propagation path. In some examples, the first edge may include a relation type indication comprising a necessity indication, a support indication, a method indication, or a relation indication. In some examples, to update the domain graph, the server 102 may further update one or more other estimated proficiency indications of one or more other skill nodes adjacent to the second skill node based on the first answer, and one or more other edges corresponding to the one or more other skill nodes. In some examples, the server 102 may iteratively, for each of multiple iterations, identify a next skill node of the plurality of skill nodes based on a potential information gain associated with the next skill node, provide a next question associated with the next skill node, receive a next answer in response to the next question, and further update the domain graph based on the next answer.

[0097] Returning again to FIG. 5, at block 510, the server 102 may provide a current proficiency indication associated with the updated domain graph. In some examples, after further updating the domain graph, the current proficiency indication may include at least one of: current estimated proficiencies corresponding to a subset of the multiple skill nodes, or an aggregate of the subset of current estimated proficiencies, the subset of the multiple skill nodes corresponding to one or more selected objectives. In some examples, the current proficiency indication may include the estimated proficiency indication or point estimate of a skill node. In other examples, the current proficiency indication may include the estimated proficiency indications or point estimates of multiple skill nodes. In particular, the multiple skill nodes may be associated with objectives (e.g., selected by an instructor). In even further examples, the current proficiency indication may include an aggregate of the estimated proficiency indications or point estimates of the multiple skill nodes. The aggregate may include an average proficiency score of the learner (e.g., for the objectives selected by the instructor). In some examples, the learner may know the proficiency score (e.g., 85%) for the Algebra I class, which includes instructor selected objectivesafter the test. Further, the current proficiency indication may further include the lowest and / or the highest estimated proficiency indications or point estimates of skill nodes in the domain graph.

[0098] In further examples, the server 102 may further determine a next skill node based on a combination of the current proficiency and resulting propagated improvements to downstream skill nodes of the plurality of skill nodes, and provide a remediation action associated with the next skill node. In some examples, the remediation action may include at least one of: one or more questions, lectures, or assignments being associated with the next skill node. In some examples, the next skill node may be determined to maximize the expected downstream proficiency gains. So, the server 102 may determine the next skill node among the multiple skill nodes in the domain graph that will maximally benefit related downstream skill nodes. That is, when the remediation action is performed associated with the next node, the learner may have the greatest improvement of the learner’s proficiency.

[0099] FIG. 14 is a flowchart illustrating an example process 1400 and technique for adaptive diagnostic assessment training, in accordance with various aspects of the techniques described in this disclosure. The flowchart of FIG. 14 utilizes various system components that are described below with reference to FIGS. 1-4 and 15. In some examples, the process 1400 may be carried out by the server(s) 102 illustrated in FIG. 3, e g., employing circuitry and / or software configured according to the block diagram illustrated in FIG. 2. In some examples, the process 1400 may be carried out by any suitable apparatus or means for carrying out the functions or algorithm described below. Additionally, although the blocks of the process 1400 are presented in a sequential manner, in some examples, one or more of the blocks may be performed in a different order than presented, in parallel with another block, or bypassed.

[0100] At block 1402, the server 102 identifies multiple skill nodes in a domain graph stored in an electronic memory. In some examples, the multiple skill nodes of the domain graph arranged as a directed acyclic graph. The skill node and the domain graph in block 1402 is substantially similar to the skill node and the domain graph in block 502 in FIG. 5. In some examples, each skill node of the multiple skill nodes may be associated with a point estimate of a distribution associated with a respective skill node. Also, a length of an edge for each skill node of the multiple skill nodes on the domain graph may be determined by subtracting a first point estimate associated with a parent skill node of a respective skill node from a second point estimate associated with the respective skill node as a child skill node of the parent skill node. In someexamples, the point estimate associated with each skill node may be reparameterized as an alpha and beta parameter pair of a Beta distribution for a respective node. Further, a sum of the alpha and beta parameter pair for each skill node may be held constant across the graph, such that a path connecting the first and second dummy nodes comprises a Dirichlet distribution. Also, the marginal distributions of the Dirichlet distribution may correspond to Beta marginal distributions of the skill nodes along the path or each chain or path from the first to the second dummy node.

[0101] At block 1404, the server 102 generates a first dummy node, indicating perfect proficiency. In some examples, the first dummy node in block 1404 may be substantially similar to the first dummy node 1002 in FIGS. 10 and 11. In some examples, the server 102 generates the first dummy node by defining a node, similar to the first dummy node 1002 of FIG. 10 or 11, and adding or saving the node to the domain graph in the electronic memory.

[0102] At block 1406, the server 102 connects the first dummy node to a root skill node of the multiple skill nodes. In some examples, the root skill node in block 1406 may be substantially similar to the root skill node 612 in FIGS. 10 and 11 or the skill node A 1102 in FIG. 11. In some examples, the server 102 connects the first dummy node to the root skill node, which is similar to the root skill node 612 in FIGS. 10 and 11 and / or the skill node A 1102. In some examples, the server 102 connects the first dummy node to the rook skill node using a root edge, which is similar to the root edge 1004 in FIGS. 10 and 11 and / or 1112 in FIG. 11. The server 102 may add or save the root edge between the first dummy node and the root skill node in the electronic memory.

[0103] At block 1408, the server 102 generates a second dummy node, indicating perfect non-proficiency. In some examples, the second dummy node in block 1408 may be substantially similar to the second dummy node 1006 in FIGS. 10 and 11. In some examples, the server 102 generates the second dummy node by defining a node, similar to the second dummy node 1006 of FIG. 10 or 11, and adding or saving the node to the domain graph in the electronic memory.

[0104] At block 1410, the server 102 connects the second dummy node to a leaf skill node of the plurality of skill nodes. In some examples, the leaf skill node in block 1410 may be substantially similar to the leaf skill node 620 in FIGS. 10 and 11 or the skill node D 1108 in FIG. 11. In some examples, the server 102 connects the second dummy node to the leaf skill node, which is similar to the leaf skill node 620 in FIGS. 10 and 11 and / or the skill node D 1108. In some examples, the server 102 connects the second dummy node to the leaf skill node using a leaf edge, which is similar to the leaf edge 1008 in FIGS. 10 and 11 and / or 1122 in FIG. 11. The server 102may add or save the leaf edge between the second dummy node and the leaf skill node in the electronic memory.

[0105] In some examples, the server 102 may further identify the longest path connecting the first and second dummy nodes and a subset of the plurality of skill nodes along the longest path or chain. Further, the server 102 may further generate multiple first synthetic profiles for the longest path (e.g., by drawing samples from a Dirichlet distribution whose parameters are defined by the edge lengths associated with that longest chain). Each synthetic profile of the multiple first synthetic profiles may include multiple first ground truth proficiency indications corresponding to the subset of the plurality of skill nodes. In such examples, the server 102 may further generate multiple second synthetic profiles for each of one or more subchains intersecting with a skill node of the subset of the multiple skill nodes (e.g., by drawing samples from the Dirichlet distribution associated with each chain, appropriately scaled based on the values previously assigned to the endpoints of each subchain). In some examples, the server 102 has already assigned the synthetic profiles to the subset of the multiple skill nodes. In some examples, the server 102 may iteratively generate multiple other synthetic profiles for each of one or more subchains intersecting with another skill node of the subset of the plurality of skill nodes. In some examples, the server 102 may at every iteration identify the subchains with the fewest edges to populate the profiles of skill nodes lying along those chains before proceeding to any longer unpopulated subchains.

[0106] FIG. 15 is a schematic diagram 1500 conceptually illustrating synthetic profiles, in accordance with various aspects of the techniques described in this disclosure. In the examples of five skill nodes shown in FIG. 10, the five skill nodes 612, 614, 616, 618, 620 with the first and second dummy nodes 1002, 1006 may be the longest path in the domain graph. In some examples, the server 102 may generate multiple synthetic profiles. Each row of the 18 rows in the diagram 1500 of FIG. 15 represents a synthetic profile. Each synthetic profile for a synthetic learner may include five ground truth proficiency indications corresponding to the five skill nodes 612, 614, 616, 618, 620, respectively. For example, a synthetic learner corresponding to synthetic profile 1520 may include five ground truth proficiency indications 1502, 1504, 1506, 1508, 1510 corresponding to the five skill nodes 612, 614, 616, 618, 620, respectively. Each ground truth proficiency indication is illustrated in FIG. 15 as a vertical line within one of the synthetic profiles. For each synthetic profile in the diagram 1500, the left most ground truth proficiency indication corresponds to the skill node 612 and, moving rightward, the second through fifth ground truthproficiency indications correspond to the skill nodes 614, 616, 618, and 620, respectively. Additionally, in the diagram 1500, each synthetic profile may have an x-axis ranging from 1.0 (on the left extreme) and 0.0 (on the right extreme), and may have a y-axis ranging from 1.0 (on the upper extreme) to 0.0 (on the lower extreme). The synthetic profiles may be randomly drawn from Dirichlet distributions such that the marginal Beta distributions associated with each skill node exhibit desired, target point estimates. Further, the server may generate other synthetic profiles for other subchains intersecting each skill node of multiple skill nodes along the longest path. Based on the multiple synthetic profiles, the server 102 may run simulations to tune the hyperparameters for the various edge types (W, X, Y, Z, R, S).

[0107] In some examples, the synthetic profiles may be generated to correspond to the prior distributions associated with the skill nodes in the domain graph in connection with block 502 of FIG. 5. The prior data may indicate, based on empirical data, the average proficiency associated with each skill node. In other words, each synthetically generated student profile includes a proficiency estimate associated with every node, and when aggregated across a population of synthetic profiles, the aggregated values will correspond to empirically observed aggregated values. Sampling from Dirichlet distributions may thus enable the generation of random student profiles, such that every marginal distribution takes the shape of a Beta distribution, the mean of which converges to a targeted point estimate.

[0108] In further examples, the pseudo-code to generate synthetic profiles may be as follows:1) If a node has no prerequisites (a so-called “root” node), connect it to a dummy “perfect 1.0 proficiency” node representing a perfect 1.0 proficiency (this ensures that all nodes are part of a Dirichlet chain).2) If a node has no postrequisites (a so-called “leaf’ node), connect it to a dummy “perfect 0.0 proficiency” node representing an impossibly difficult skill (again, this ensures that all nodes are part of a Dirichlet chain).3) Find the longest chain on the graph connecting the two dummy nodes.4) Populate that longest chain first by treating it as a Dirichlet distribution whose parameters are determined based on the point estimates communicated by the priors associated with the nodes along that chain.5) Iteratively populate all the other subchains that intersect, at any point, with the nodes that have already been populated, always starting with the shortest subchains.6) Exploit the neutrality and aggregation properties of the Dirichlet by treating each new subchain as an independent Dirichlet distribution, re-normalizing each to ensure that each subchain begins and ends as the values previously assigned to those beginning and ending skills nodes, with the intermediate edges on the subchain linearly rescaled to ensure that all full edge chains sum to one.7) Once all nodes on the graph have been populated, validate the entire graph to ensure that each sample remains consistent with the graph’s structure (prerequisite proficiencies are always higher than their postrequisites). If any inconsistencies are discovered, those inconsistencies may either be ignored, the entire sample may be rejected and regenerated from scratch, or the offending subchains may be resampled until the inconsistency disappears.

[0109] Other examples and uses of the disclosed technology will be apparent to those having ordinary skill in the art upon consideration of the specification and practice of the invention disclosed herein. The specification and examples given should be considered exemplary only, and it is contemplated that the appended claims will cover any other such embodiments or modifications as fall within the true scope of the invention.

[0110] The Abstract accompanying this specification is provided to enable the United States Patent and Trademark Office and the public generally to determine quickly from a cursory inspection the nature and gist of the technical disclosure and in no way intended for defining, determining, or limiting the present invention or any of its embodiments.

Claims

CLAIMSWhat is claimed:

1. A method for an adaptive diagnostic assessment, comprising: determine, by an electronic processor, a first skill node of a plurality of skill nodes in a domain graph stored in an electronic memory, the plurality of skill nodes of the domain graph arranged as a directed acyclic graph, the first skill node: comprising a first estimated proficiency indication as a continuous marginal probability distribution, being associated with one or more questions, and being connected to one or more adjacent skill nodes of the plurality of skill nodes via one or more edges, the one or more adjacent skill nodes including a second skill node that comprises a second estimated proficiency indication and that is connected to the first skill node via a first edge of the one or more edges; providing, by the electronic processor, a first question of the one or more questions to a learner; receiving, by the electronic processor, a first answer from the learner for the first question; updating, by the electronic processor, the domain graph based on the first answer by: updating the first estimated proficiency indication of the first skill node, and updating the second estimated proficiency indication of the second skill node based on the first answer and the first edge; and providing, by the electronic processor, a current proficiency indication associated with the updated domain graph.

2. The method of claim 1, wherein the determining of the first skill node comprises: for each skill node of the plurality of skill nodes corresponding to the one or more of questions: determining a plurality of gains between a plurality of current point estimates corresponding to a plurality of other skill nodes than a respective skill node in the plurality of skillnodes and a plurality of hypothetical point estimates corresponding to the plurality of other skill nodes if an answer to a respective question associated with the respective skill node is correct and incorrect; and determining a potential information gain for the respective skill node based on the plurality of gains, and an estimated proficiency indication of the respective skill node; and determining the first skill node corresponding to a maximum gain based on the potential information gain for each skill node of the plurality of skill nodes.

3. The method of claim 2, wherein the determining of the plurality of gains of each skill node of the plurality of skill nodes comprises: for each skill node of the plurality of skill nodes corresponding to the one or more questions: determining a plurality of first gains of the plurality of gains between a plurality of first current point estimates corresponding to the plurality of other skill nodes and a plurality of first hypothetical point estimates corresponding to the plurality of other skill nodes if the answer to the respective question associated with the respective skill node is correct; and determining a plurality of second gains of the plurality of gains between a plurality of second current point estimates corresponding to the plurality of other skill nodes and a plurality of second hypothetical point estimates corresponding to the plurality of other skill nodes if the answer to the respective question associated with the respective skill node is incorrect; wherein the determining of the potential information gain for each skill node comprises: for each skill node of the plurality of skill nodes corresponding to the one or more questions: determining the potential information gain for the respective skill node based on: a point estimate associated with the respective node multiplied by a first sum of the plurality of first gains, and an estimated non-proficiency indication associated with the respective node multiplied by a second sum of the plurality of second gains.

4. The method of claim 1, wherein the determining of the first skill node comprises: receiving one or more selected objectives corresponding to a subset of the plurality of skill nodes; identifying a subset of current point estimates corresponding to the subset of the plurality of skill nodes;determining a subset of hypothetical point estimates corresponding to the subset of the plurality of skill nodes; and determining the first skill node based on the subset of current point estimates and the subset of hypothetical point estimates.

5. The method of claim 4, wherein the determining of the first skill node further comprises: for each skill node of the plurality of skill nodes corresponding to the one or more questions, determining a potential information gain for a respective skill node of the plurality of skill nodes based on an estimated proficiency indication of the respective skill node and one or more gains corresponding to the subset of the plurality of skill nodes between the subset of current point estimates and the subset of hypothetical point estimates, wherein the determining of the first skill node based on the subset of current point estimates and the subset of hypothetical point estimates comprises: determining the first skill node corresponding to a maximum gain based on the potential information gain for each skill node of the plurality of skill nodes.

6. The method of claim 1, wherein the one or more questions of the first skill node correspond to a same level of difficulty.

7. The method of claim 1, wherein a marginal probability distribution for each skill node is maintained or calculated instead of maintaining or calculating a joint probability distribution across the domain graph.

8. The method of claim 7, wherein the continuous marginal probability distribution comprises a Beta distribution representing a distribution over possible proportions of the one or more questions that the learner is expected to answer correctly, and wherein the Beta distribution is parameterized with: an alpha parameter representing a first number of correct answers at the first skill node, anda beta parameter representing a second number of incorrect answers at the first skill node.

9. The method of claim 8, wherein the first estimated proficiency indication further comprises a point estimate indication of the continuous marginal probability distribution.

10. The method of claim 9, wherein a root skill node of the plurality of skill nodes without a parent node is connected by a root edge to a first dummy node having a perfect proficiency indication, and wherein a leaf skill node of the plurality of skill nodes without a child node is connected by a leaf edge to a second dummy node having a perfect non-proficiency indication.

11. The method of claim 10, wherein a path connecting the first dummy node to the second dummy node comprises a chain of edges, and wherein lengths corresponding to the edges of the chain sum to the perfect proficiency indication.

12. The method of 11, wherein the chain of the edges includes the first edge, and wherein the point estimate indication associated with the first skill node comprises a probability being the perfect proficiency indication subtracting a sum of first lengths of first edges in the chain connecting the first skill node to the first dummy node.

13. The method of claim 11, wherein the chain of the edges includes the first edge, and wherein the point estimate indication associated with the first skill node comprises a probability being a sum of second lengths of second edges in the chain connecting the first skill node to the second dummy node.

14. The method of claim 13, wherein the updating of the second estimated proficiency indication of the second skill node comprises: updating the lengths of the edges in the chain to be the perfect proficiency indication after updating the second estimated proficiency indication of the second skill node.

15. The method of claim 8, wherein the updating of the first skill node comprises, in response to the first answer being correct, updating the alpha parameter of the Beta distribution by incrementing the first number of correct answers of the alpha parameter; and in response to the first answer being incorrect, updating the beta parameter of the Beta distribution by incrementing the second number of incorrect answers of the beta parameter.

16. The method of claim 8, wherein the updating of the first skill node comprises, in response to the first answer being correct, updating the alpha parameter of the Beta distribution by adding a constant hyperparameter to the alpha parameter; and in response to the first answer being incorrect, updating the beta parameter of the Beta distribution by adding the constant hyperparameter to the beta parameter.

17. The method of claim 1, further comprising: determining the first estimated proficiency indication of the first skill node based on a prior distribution of the first skill node and a likelihood function of the first skill node, wherein the prior distribution is a Beta distribution and the likelihood function is binomial.

18. The method of claim 1, wherein the first edge comprises a relation type indication comprising a necessity indication, a support indication, a method indication, or a relation indication.

19. The method of claim 1, wherein the second estimated proficiency indication of the second skill node is updated based on at least one of: a correctness of the first answer, a relation type indication, a propagation direction, or a propagation path.

20. The method of claim 1, wherein the updating of the domain graph based on the first answer further comprises: updating one or more other estimated proficiency indications of one or more other skill nodes adjacent to a second skill node based on the first answer, and one or more other edges corresponding to the one or more other skill nodes.21 . The method of claim 1 , further comprising: iteratively, for each of a plurality of iterations: identifying, by the electronic processor, a next skill node of the plurality of skill nodes based on a potential information gain associated with the next skill node; providing, by the electronic processor, a next question associated with the next skill node; receiving a next answer in response to the next question; and further updating, by the electronic processor, the domain graph based on the next answer.

22. The method of claim 21, wherein after further updating the domain graph, the current proficiency indication comprises at least one of: current estimated proficiencies corresponding to a subset of the plurality of skill nodes, or an aggregate of the subset of current estimated proficiencies, the subset of the plurality of skill nodes corresponding to one or more selected objectives.

23. The method of claim 1, further comprising: determining a next skill node based on a combination of the current proficiency and resulting propagated improvements to downstream skill nodes of the plurality of skill nodes; and providing a remediation action associated with the next skill node, the remediation action comprises at least one of: one or more questions, lectures, or assignments being associated with the next skill node.

24. A method for adaptive diagnostic assessment training, comprising: identifying, by an electronic processor, a plurality of skill nodes in a domain graph stored in an electronic memory, the plurality of skill nodes of the domain graph arranged as a directed acyclic graph; generating a first dummy node, indicating perfect proficiency; connecting the first dummy node to a root skill node of the plurality of skill nodes; generating a second dummy node, indicating perfect non-proficiency; and connecting the second dummy node to a leaf skill node of the plurality of skill nodes.

25. The method of claim 24, wherein each skill node of the plurality of skill nodes is associated with a point estimate of a distribution associated with a respective skill node, and wherein a length of an edge for each skill node of the plurality of skill nodes on the domain graph is determined by subtracting a first point estimate associated with a parent skill node of a respective skill node from a second point estimate associated with the respective skill node as a child skill node of the parent skill node.

26. The method of claim 25, wherein the point estimate associated with each skill node is reparameterized as an alpha and beta parameter pair of a Beta distribution for a respective node, wherein a sum of the alpha and beta parameter pair for each skill node is constant across the graph, such that a path connecting the first and second dummy nodes comprises a Dirichlet distribution, and wherein marginal distributions of the Dirichlet distribution correspond to Beta marginal distributions of skill nodes along the path from the first dummy node to the second dummy node.

27. The method of claim 26, further comprising: identifying a longest path connecting the first and second dummy nodes and a subset of the plurality of skill nodes along the longest path.

28. The method of claim 27, further comprising: generating a plurality of first synthetic profiles for the longest path, each synthetic profile of the plurality of first synthetic profiles comprising a plurality of first ground truth proficiency indications corresponding to the subset of the plurality of skill nodes.

29. The method of claim 28, further comprising: generating a plurality of second synthetic profiles for each of one or more subchains intersecting with a skill node of the subset of the plurality of skill nodes.

30. The method of claim 29, further comprising: iteratively generating a plurality of other synthetic profiles for each of one or more subchains intersecting with another skill node of the subset of the plurality of skill nodes.

Images