Hybrid human-computer learning system

The hybrid human-computer learning system addresses the inefficiencies of existing expert systems by aggregating expert opinions through a neural network architecture, enhancing the speed and accuracy of generating actionable insights.

JP2024164021AInactive Publication Date: 2024-11-26ANTHROP LLC

Patent Information

Application Number
JP2024121910
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2020-03-31
Filing Date
2024-07-29
Publication Date
2024-11-26
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

Existing expert systems struggle to encode unstructured or unverbalized human knowledge into machine-readable formats efficiently, leading to difficulties in providing real-time, accurate, and up-to-date expert opinions due to the complex and time-consuming nature of knowledge extraction.

Method used

A hybrid human-computer learning system that aggregates subjective and objective answers from multiple experts, utilizing a neural network architecture to sort and rank solutions based on confidence scores, enabling real-time generation of actionable insights.

Benefits of technology

The system effectively distills multiple expert opinions into clear answers, improving the speed, accuracy, and cost-effectiveness of providing expert opinions on complex or subjective tasks.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024164021000001_ABST
    Figure 2024164021000001_ABST
Patent Text Reader

Abstract

To provide a system, apparatus and method related to an expert learning system which copes with one or more of defects of a known expert system.SOLUTION: A human-computer hybrid learning system comprises a series of interconnected layers of nodes, and each node 114 comprises a device 118 communicating with a human (either a user or an expert) 116. The device receives an input and provides an output to a task server. The device has data for operations stored in the device, such as task data 120, expert data 122, and network data 124 and utilizes an application 119 installed in the device. The application registers the human 116 as an expert to allow the human to participate in the resolution of a task and registers the human 116 as a user to, for example, allow the human to submit his or her task.SELECTED DRAWING: Figure 4
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to a hybrid human-computer learning system. [Background technology]

[0002] An expert system is a computer system programmed to emulate reasoning tasks by interpreting the coded knowledge of human experts. Existing expert systems are static, offline systems designed to be updated with the experts' knowledge, either once or periodically. Such expert systems typically use a static database of the experts' knowledge to recommend decisions and provide advice in areas such as medical diagnosis, stock trading, art, music, or movie evaluation, and other subjective or objective domains. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] U.S. Pat. No. 9,471,883 Summary of the Invention [Problem to be solved by the invention]

[0004] Existing examples of expert systems attempt to encode unstructured or unverbalized knowledge into a machine-readable format and emulate human emotions and subjective reasoning, as well as process the knowledge into actionable insights in real-time. In existing solutions, extracting knowledge from human experts and encoding it into a machine-readable format is nearly impossible to perform in real-time due to the complex, tedious, and time-consuming nature of existing expert systems. Therefore, existing systems generally fail to provide accurate and up-to-date expert opinion on any topic. [Means for solving the problem]

[0005] SUMMARY The present disclosure provides systems, apparatus, and methods for an expert learning system, and addresses one or more of the shortcomings of known expert systems discussed above.

[0006] In some embodiments, the system as a data processing system for providing a solution to a user provided task may comprise a memory, one or more processors, and a plurality of instructions stored in the memory and executable by the one or more processors. The instructions are enabled to receive task-related data corresponding to a selected task, the task being identified by a selected one of one or more domains of expertise, and automatically select from a plurality of experts a first subset of experts associated with the selected domain and a second subset of experts associated with the selected domain. The instructions are further enabled to communicate the task-related data to a plurality of first electronic devices, each of the first electronic devices being associated with one of the respective experts of the first subset of experts, and receive, via the first electronic device, from each expert of the first subset of experts a respective task solution and an associated first confidence score. The instructions are further enabled to: generate a first set of task solutions sorted by the first confidence score based on the task solutions received from the first subset of experts; and communicate the task related data and the first set of task solutions to a plurality of second electronic devices, each of the second electronic devices being associated with a respective one of the experts in the second subset of experts. The instructions are further enabled to receive, via the second electronic device, information from each expert in the second subset of experts indicative of a respective selected solution selected from the first set of task solutions and an associated second confidence score, generate a second set of task solutions sorted by the second confidence score based on the information indicative of the selected solutions, generate a preliminary output based on the second set of task solutions, and communicate a final output to a user interface based on the preliminary output.

[0007] In some embodiments, a data processing system for providing a solution to a user provided task may include a memory, one or more processors, and a plurality of instructions stored in the memory and executable by the one or more processors, the instructions being enabled to: receive task-related data corresponding to a selected task, the selected task being identified by a selected domain of one or more domains of expertise, and automatically select, from a plurality of experts, a first expert subset, one or more intermediate expert subsets, and a final expert subset, each of the first expert subset, one or more intermediate expert subsets, and the final expert subset associated with the selected domain. The instructions are further enabled to: transmit task-related data to a plurality of first electronic devices, each of which is associated with a respective one of the experts in a first expert subset; receive via the first electronic devices from each of the experts in the first expert subset a respective task solution and an associated first confidence score; and generate an intermediate task solution set sorted by first confidence score based on the task solutions received from the first expert subset. The instructions are further enabled to: transmit, for each of one or more intermediate expert subsets, successively, the task-related data and the intermediate task solution set to a plurality of electronic devices, each of which is associated with a respective one of the experts in a respective intermediate expert subset; receive via the electronic device from each expert in a respective intermediate expert subset a respective selected solution selected from the intermediate task solution set and an associated intermediate confidence score; and update the intermediate task solution set sorted by intermediate confidence score based on the information indicative of the selected solution.The instructions are further enabled to communicate the intermediate task-related data to a plurality of final electronic devices, each of the final electronic devices being associated with a respective one of the experts in the final expert subset; receiving information from each expert in the final expert subset via the final electronic device indicative of a respective selected solution selected from the intermediate task solution set and an associated final confidence score; and communicating to the user interface the final task solution set based on the information indicative of the selected solutions.

[0008] The features, functions, and advantages may be achieved independently in various embodiments of the present disclosure or may be combined in yet other embodiments, further details of which can be seen by reference to the following description and drawings. [Brief description of the drawings]

[0009] [Figure 1] Schematic diagram of an exemplary neural network model. [Diagram 2] 1 is a schematic diagram of an exemplary learning system according to an aspect of the present disclosure. [Diagram 3] 3 is a schematic diagram of the learning system of FIG. 2, further illustrating the input layer, hidden layer(s), and output layer. [Figure 4] 3 is a schematic diagram of a node of the learning system of FIG. 2 according to an aspect of the present disclosure. [Diagram 5] FIG. 3 is a schematic diagram showing data flow in the learning system of FIG. 2. [Figure 6] Schematic of the input layer in Figure 3. [Figure 7] Schematic of the hidden layer(s) in Figure 3. [Figure 8] Schematic of the output layer in Figure 3. [Figure 9] 1 is a flowchart depicting steps of an exemplary method of operation of a task server according to an aspect of the present disclosure. [Figure 10] 4 is a flow chart illustrating steps of an exemplary method for training a learning system according to an aspect of the present disclosure. [Figure 11] 1 is a schematic diagram of an exemplary data processing system in accordance with an aspect of the present disclosure. [Figure 12] 1 is a schematic diagram of an exemplary distributed data processing system in accordance with an aspect of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Various aspects and examples of the learning system (or expert system) are described below and illustrated in the associated drawings. Unless specifically excluded, the learning system and / or various components thereof according to the present teachings may comprise at least one of the structures, components, functionality, and / or variations described, illustrated, and / or incorporated herein. Furthermore, unless specifically excluded, the process steps, structures, components, functionality, and / or variations described, illustrated, and / or incorporated herein in connection with the present teachings may be included in other similar devices and methods, including interchangeable among the disclosed embodiments. The following description of various examples is merely exemplary in nature and is not intended to limit the disclosure, its application, or uses in any way. Furthermore, the advantages provided by the examples and embodiments described below are exemplary in nature, and not all examples and embodiments may provide the same advantages or to the same extent of advantages.

[0011] This detailed description includes the following sections, which immediately follow: (1) Definitions, (2) Overview, (3) Examples, Components, and Alternatives, (4) Advantages, Features, and Benefits, and (5) Conclusion. The Examples, Components, and Alternatives section is further divided into subsections, labeled A through F.

[0012] [Definition] In this document, the following definitions apply unless otherwise stated. "Comprising," "comprising," "including," and "having" (and conjugations thereof) are used interchangeably to mean including, but not necessarily limited to, and are open-ended terms not intended to exclude additional, unrecited elements or method steps.

[0013] The terms "first," "second," and "third," etc. are used to distinguish or distinguish various members of a group, and are not intended to denote sequential or numerical limitations.

[0014] "AKA" may be used to mean "also known as" as well as to indicate alternative or corresponding terms for a given element or elements. "Processing logic" describes any suitable device or hardware configured to process data by performing one or more logical and / or arithmetic operations (e.g., executing coded instructions). For example, processing logic may comprise one or more processors (e.g., a central processing unit (CPU) and / or a graphics processing unit (GPU)), microprocessors, clusters of processing cores, FPGAs (field programmable gate arrays), artificial intelligence (AI) accelerators, digital signal processors (DSPs), and / or any other suitable combination of logic hardware.

[0015] A "controller" or "electronic controller" comprises processing logic programmed with instructions to perform control functions with respect to control elements. For example, an electronic controller may be configured to receive input signals and compare the input signals with selected control values ​​or set points as well as determine output signals to the control elements (e.g., motors or actuators) to provide corrective action based on the comparison. In another example, an electronic controller may be configured to interface between a host device (e.g., desktop computer, mainframe, etc.) and a peripheral device (e.g., memory device, input / output device, etc.) as well as control and / or monitor input and output signals to and from the peripheral device.

[0016] "Providing" in the context of a method can include receiving, obtaining, purchasing, manufacturing, producing, processing, pre-processing, and / or the like, such that the provided objects or materials are in a condition and configuration for performing other steps.

[0017] In this disclosure, one or more publications, patents, and / or patent applications may be incorporated by reference. However, such material is incorporated only to the extent that there is no conflict between the incorporated material and the descriptions and drawings set forth herein. In the event of any such conflict, including any conflict of terminology, the present disclosure will control.

[0018] [overview] In general, the disclosed hybrid human-computer learning system provides an architectural framework for an expert to analyze questions and / or problems (e.g., tasks) submitted by a user. The system enables aggregation of subjective and objective answers from multiple answers provided by experts, while providing improved performance in terms of speed, accuracy, and cost (e.g., time, money, etc.). For example, medical personnel can utilize the system to receive medical diagnoses for rare or complex medical conditions through a network of physicians.

[0019] The learning system of the present disclosure utilizes networks modeled after deep learning, recurrent neural networks, as well as experts to solve subjective and difficult problems. The learning system may provide solutions that would otherwise be impossible for a computer to solve through known methods. As with standard neural networks, the system can also be used for predictive modeling.

[0020] The learning system is configured to allow one or more skilled experts to provide their own solutions to a proposed challenge along with an associated confidence score, as well as submit the solutions to a voting group of experts, each of which selects the correct or most correct option from among the provided solutions based on the expert's opinion, as well as providing its own confidence score for that selection. Further layers and / or various examples of layer orderings may be utilized, as well as the system being configured to output an answer, for example, in the form of a list of solutions sorted by confidence.

[0021] Disclosed herein is a technical solution for providing one or more expert-provided solutions to a user-provided task. Specifically, the disclosed system and method address a technical problem that arises in the field of machine learning coupled with learning systems and other neural network techniques: providing meaningful solutions to subjective tasks, such as tasks for which a clear, objective solution may not exist and / or tasks that lack a clearly defined training set. In other words, the learning system described herein implements a practical solution that has the unique ability to distill the opinions and subjective answers of multiple experts into a clear answer to a user-provided question or task. The computer-based system and method disclosed herein provide an improved solution to this technical problem by incorporating the subjective answers and confidence rankings of subject matter experts into a network configured to automatically identify the optimal solution.

[0022] The disclosed systems and methods provide an integrated and practical application of the principles discussed herein. In particular, the systems and methods described herein are an improvement over known learning systems because they generate one or more subjective solutions to tasks that may not have an objective answer. Additionally, the disclosed systems and methods describe specific aspects that provide task solutions that rely on an expert-provided review process and a recurrent scoring system. Thus, the disclosed systems and methods also utilize the relevant principles in a meaningful and limited way.

[0023] Aspects of the learning system may be embodied as a computer method, a computer system, or a computer program product. Accordingly, aspects of the learning system may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, microcode, etc.), or an embodiment combining software and hardware aspects, all of which may be referred to generally herein as a "circuit," "module," or "system." Additionally, aspects of the learning system may take the form of a computer program product embodied in a computer readable medium (or medium) having computer readable program code / instructions embodied on the computer readable medium (or medium).

[0024] Any combination of computer readable media may be utilized. The computer readable medium may be a computer readable signal medium and / or a computer readable storage medium. The computer readable storage medium may comprise an electronic, magnetic, optical, electromagnetic, infrared, and / or semiconductor system, apparatus, or device, or any suitable combination thereof. More specific examples of computer readable storage media may include an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read only memory (ROM), an erasable programmable read only memory (EPROM or flash memory), an optical fiber, a portable compact disk read only memory (CD-ROM), an optical storage device, a magnetic storage device, and / or any suitable combination thereof. In the context of the present disclosure, the computer readable storage medium may comprise any suitable non-transitory tangible medium that comprises or can store (store) a program for use by or in connection with an instruction execution system, apparatus, or device.

[0025] A computer-readable signal medium may comprise a propagated data signal in which computer-readable program code is embodied, for example in baseband or as part of a carrier wave. Such a propagated signal may take any of a variety of forms, including, but not limited to, electromagnetic, optical, and / or any suitable combination thereof. A computer-readable signal medium is not a computer-readable storage medium, but may comprise any computer-readable medium that can communicate, propagate, or carry a program for use by or in connection with an instruction execution system, apparatus, or device.

[0026] The program code embodied in the computer readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, fiber optic cable, RF, etc., and / or any suitable combination thereof.

[0027] Computer program code for performing operations for aspects of the learning system may be written in one or any combination of programming languages, including object-oriented programming languages ​​(e.g., Java, C++, etc.), traditional procedural programming languages ​​(e.g., C), and functional programming languages ​​(e.g., Haskell). Mobile apps may be developed using any suitable language, including those mentioned above, as well as Objective-C, Swift, C#, HTML5, etc. The program code may run entirely on the user's computer, partially on the user's computer and as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), and / or the connection may be made to an external computer (e.g., via the Internet using an Internet Service Provider).

[0028] Aspects of the learning system may be described below with reference to flowchart illustrations and / or block diagrams of methods, apparatus, systems, and / or computer program products. Each block and / or combination of blocks in the flowcharts and / or block diagrams may be implemented by computer program instructions. The computer program instructions may be programmed or otherwise provided to processing logic (e.g., a processor of a general purpose computer, a special purpose computer, a field programmable gate array (FPGA), or other programmable data processing device) to produce a machine as well as to create means for performing the function / acts specified in the block(s) of the flowcharts and / or block diagrams by instructions (e.g., machine readable) executing via the processing logic.

[0029] Additionally or alternatively, these computer program instructions may be stored on a computer-readable medium that can direct processing logic and / or any other suitable device to function in a particular manner, such that the instructions stored on the computer-readable medium produce an article of manufacture comprising instructions that perform the functions / acts identified in the flowchart and / or block diagram block(s).

[0030] The computer program instructions may also be loaded into processing logic and / or any other suitable device to cause a series of operational steps to be performed on the device to generate a computer-implemented process such that the executed instructions provide a process for performing the function / act specified in the flowchart and / or block diagram block(s).

[0031] Any flowcharts and / or block diagrams in the drawings are intended to illustrate the architecture, functionality, and / or operation of possible implementations of systems, methods, and computer program products according to aspects of the learning system. In this regard, each block may represent a module, segment, or portion of code constituting one or more executable instructions for performing a specified logical function(s). In some implementations, the functions noted in the blocks may occur in a different order than that noted in the drawings. For example, two blocks shown in succession may in fact be executed substantially simultaneously, and the blocks may sometimes be executed in the reverse order, depending on the functionality involved. Each block and / or combination of blocks may be implemented by a special purpose hardware-based system (or a combination of special purpose hardware and computer instructions) that performs the specified functions or acts.

[0032] [Examples, components, and alternatives] The following sections describe selected aspects of an exemplary learning system, as well as related systems and / or methods. The examples in these sections are for illustrative purposes and should not be construed as limiting the scope of the disclosure. Each section can comprise one or more different embodiments or examples, and / or contextual or related information, functionality, and / or structure.

[0033] A. An Exemplary Hybrid Human-Computer Learning System As shown in FIGS. 1-3, this section describes an exemplary hybrid human-computer learning system 100, which is one example of the learning systems described above.

[0034] The learning system 100 identifies meaningful solutions to user-given tasks using a framework similar in structure and function to neural networks. In general, neural networks are computing systems inspired by and similar to biological neural networks found, for example, in animal brains. Such systems not only learn to solve such tasks by considering examples of the tasks and learning to arrive at known solutions to those examples, but also generally have no task-specific rules. Some neural networks can learn autonomously in an unsupervised manner without examples, such as drawing conclusions from complex and seemingly unrelated collections of information.

[0035] With reference to Figure 1, neural networks are typically based on a collection of connected units (commonly called nodes or neurons), which may loosely model the neural structure of a biological brain. Each connection can send signals to other nodes, like synapses in a biological brain. A node that receives a signal can process the signal as well as send the signal to nodes that are connected to it.

[0036] Nodes are usually arranged or aggregated into layers, e.g., I1 to I N An input layer consisting of input nodes labeled H 1 1~H 1 N one or more hidden layers, each of which includes a first hidden layer of first hidden nodes labeled O1 through O2; N and an output layer consisting of output nodes labeled as . An input signal is provided to each input node of the input layer, for example by a user. After the signal is processed by each node of the input layer, it is propagated to each node of the first hidden layer. This signal propagation and processing can continue through the network to a final output layer. The output layer includes one or more nodes that, after processing the signal, output one or more signals as a final output. The output may be provided to a user. Additionally or alternatively, the output may be provided again to the system, i.e., as an input. If a signal traverses a layer multiple times (e.g., from an output back to an input), the learning system 100 may be referred to as a recurrent neural network. Furthermore, if a neural network includes multiple hidden layers, it may be referred to as "deep", e.g., the term deep learning.

[0037] In a typical neural network implementation, the signals propagated through each node's connections are real. When computing an aggregate of its inputs, each node utilizes a nonlinear, differentiable function (a.k.a., activation function) with respect to the sum of its inputs. In some examples, the activation function is monotonic and sigmoidal. In many neural network implementations, each node has an associated weight. Typically, each weight value is a real number. The weight can represent the effective influence or strength of the respective node in the network. For example, during processing, a node can multiply its real input by its associated weight when computing the output. A node with a large weight can have more influence on the overall network output than a node with a small weight. Some neural networks can learn autonomously (e.g., unsupervised learning), while others need to be trained (e.g., with a labeled dataset) in order to update the weights or modify the network architecture to create an optimal mapping from inputs to outputs (e.g., supervised learning). Training a neural network may comprise the use of optimization algorithms (eg, backpropagation) to find an optimal or suboptimal set of weights.

[0038] Because experts are selected to perform tasks based on prior experience, network training is not necessary for the learning system 100 to function. However, network training using data with known outcomes as targets may be utilized to improve performance and / or accuracy for particular networks and tasks (e.g., those that require study and / or training). A more detailed discussion of training the learning system is provided below in Section C.

[0039] 2 and 3, the learning system 100 utilizes an expert network 106 having a neural network architecture, where each node of the network is configured with an expert (a.k.a. a subject matter expert (SME)) and an associated device (e.g., a mobile device). The signal propagated through each node connection is the qualitative and / or quantitative analysis provided by the expert of the preceding node.

[0040] Users 102 communicate with the learning system 100 through a task server 104. The task server 104 is an example of a data processing system (see section D below). The task server 104 includes a memory device that holds a database with task data 120, expert data 122, and network data 124. All information stored in the database may be encrypted / obfuscated to ensure best security practices.

[0041] Task data 120 may comprise any data related to the task, including, for example, a description of the task (e.g., questions, explanations, media - photos, videos, etc., areas of experts appropriate for the task, and / or any other additional information related to the task). In some examples, task data 120 comprises a maximum task time limit. In some examples, the task time limit may have a default value of 5 minutes. This may be useful for time-sensitive tasks (e.g., medical diagnosis) where all experts at a particular layer may need to respond before network errors are calculated, weights are adjusted, and outputs are propagated.

[0042] In some examples, the task data 120 comprises a description of a reward. The reward may be paid to an expert based on accuracy and / or responsiveness, among other factors. Depending on the reward, an expert may earn less reward (or no reward at all) if the task is not completed in a timely manner, if the task does not include certain information, such as references if appropriate, or for a variety of other reasons. Additionally or alternatively, rewards can be provided to individual experts or to groups of experts. Additionally or alternatively, rewards may be requested by the user 102, allowing experts to be paid to participate in a task.

[0043] Expert data 122 may comprise any suitable data related to the experts, such as a unique ID for each expert, a categorical description of each expert's expertise (e.g., medicine, music, law, etc.), and / or historical data related to previous task solutions that each expert may have provided.

[0044] In some examples, the expert data 122 includes a weight value associated with each expert. In the learning system 100, the weights can represent speed and accuracy of past results / performance. Similar to the discussion above, the weights in the learning system 100 represent effective influence or strength in the expert network 106. The expert weights are utilized by the learning system 100 to amplify or attenuate inputs, increasing or decreasing the strength of the signal at a connection, thereby assigning importance to the inputs with respect to the task the algorithm is trying to learn.

[0045] The expert data 122 may further comprise a confidence level associated with each expert. An expert may rate the confidence level of its output, for example, by a value between -1 and 1, a value between 0 and 1, a value between 1 and 100, or a number of stars, or other rating system. If the confidence is too low, the output may not propagate to other nodes in the network (see discussion of activation thresholds below). Additionally, in some embodiments, experts rate the outputs of other experts.

[0046] In some examples, the task server 104 utilizes the expert data 122 to automatically classify experts based on, for example, expertise, demographics, accuracy, availability, response time, cost, etc., such that a network configuration can be determined and stored in the network data 124. In this manner, a network of registered on-call experts is created and maintained by the learning system 100.

[0047] In some examples, the network data 124 includes a learning rate value for automatically adjusting the weights, for example during the training process. In some examples, the learning rate has a default value of 0.25%. The network error is multiplied by the learning rate and the weights are adjusted accordingly. In some embodiments, the training system 100 utilizes simulated annealing, so that the learning rate decreases as the training data set is processed.

[0048] In some examples, the network data 124 comprises a maximum reentry value that indicates the maximum number of times (i.e., epochs) that the network can feed an output back to the input of the network. Returning the output to the input may help the network form a consensus (i.e., converge). In some examples, the maximum reentry value may have a default of 2 epochs.

[0049] In some examples, the network data 124 includes a minimum comment value that defines the minimum number of characters that must be typed in the notes section by each expert before propagating the output to the next layer. In some examples, the minimum comment value has a default of 20 characters. The minimum comment value may encourage experts to contribute more to their output and the overall task solution. Comments can include references that support their output.

[0050] In some examples, the network data 124 comprises an activation threshold. If an expert ranks its confidence level below the activation threshold, its output does not propagate (e.g., can be prevented from propagating) to the next layer of experts. The activation threshold is set to any value between 0% and 100%, and in some examples, may have a default value of 0.1%. If no output from other experts located in the same layer is available to be sent to the next layer, the activation threshold may be ignored, so that the output may be sent to the next layer.

[0051] In some examples, the network data 124 includes a replacement rate. Experts that provide erroneous outputs or have low weights due to poor performance may be automatically replaced by the learning system 100. The replacement rate specifies the probability that an expert may be replaced. In some examples, the replacement rate may have a default value of 0.1%. The weights of all experts in the network are normalized between 0 and 1, and if a particular expert's weight is less than the replacement rate, the expert may be removed and replaced with another expert (i.e., one that may provide more accurate results).

[0052] Before a task is processed by the expert network 106, the experts may be grouped into an input layer 108, a hidden layer 110, or an output layer 112. The experts may be automatically / dynamically selected and grouped by the task server 104 and / or the experts may be specified by a user. The experts may be ranked by their weight, which may be derived from the accuracy of their previous output. Experts with the highest weight may be placed higher (i.e., earlier) in the network, while experts with lower weights may be placed lower (i.e., later) in the network. A particular layer of the learning system may operate according to layer-specific rules, as described in more depth below. In the current example, a single hidden layer is discussed for brevity and simplicity, but the learning system 100 may alternatively utilize an expert network with more or fewer hidden layers.

[0053] Turning to FIG. 4, the user 102 and each expert of the expert network 106 are examples of typical nodes 114 of the network. Each node 114 is configured with a device 118 that communicates with a human 116 (either a user or an expert). In some embodiments, the device 118 is a smartphone. In some embodiments of the learning system 100, the device 118 comprises a desktop computer or a laptop computer. The device 118 is configured to receive input and provide output to the task server 104. Communication between the human 116 and the device 118 can be accomplished using any suitable human-machine interface (HMI), such as a graphical user interface (GUI). The device 118 may have data for the operation stored on the device 118, such as task data 120, expert data 122, and network data 124. In some examples, the device of a particular node has partial data stored thereon, so that, for example, a particular expert node may only have task data stored on its device.

[0054] Some embodiments of the learning system 100 utilize an app 119 (i.e., a software application) installed on the device 118. The app 119 is configured to allow humans 116 to register as experts and thereby participate in solving a task. In some examples, the app 119 is configured to allow humans 116 to register as users, e.g., to submit their own tasks. Through use of the app 119, experts may receive alerts, push notifications, text messages, and / or the like providing notification of new tasks for participation (e.g., in exchange for a fee). In some examples, the app 119 is configured to allow users 102 to capture photos or videos on the device 118, e.g., to submit with the new task.

[0055] Human 116 may optionally share availability and location information, e.g., anonymously. This option may be disabled by default, and if intentionally enabled by human 116, device 118 may record locations visited by human 116 along with associated timestamps. In some examples, this location data may be used to assist law enforcement agencies, for example, in solving crimes or locating missing persons.

[0056] Turning to FIG. 5, a user 102 submits a task 103 (and associated task data 120, discussed above) to a task server 104. After the task 103 is received by the task server 104, the task server estimates the cost and time for completion of the task. The task server 104 may attempt to identify specific experts who are currently available (or may become available) to join the current expert network, as well as contacting each expert's device (via push notification, text message, email, etc.). The task server 104 may then optionally perform pre-processing 105 operations on the task 103. Otherwise, the task 103 is dispatched to the expert network 106.

[0057] When utilized, pre-processing 105 is configured to tailor the learning system 100 for a particular domain or field of expertise. In one example, pre-processing 105 may be utilized to assist a music artist in receiving feedback on their music. In this example, pre-processing 105 may involve compressing or splitting audio files, or performing genre and style classification, artist identification, or acoustic analysis prior to submitting the files and tasks. In another example, pre-processing 105 may utilize a natural language processing (NLP) module on news articles and analyst opinions on stocks or other securities. In this example, a portion of text may be submitted to the learning system 100 for semantic analysis to receive a consensus on whether the price of a stock should go up or down. In yet another example, pre-processing 105 may be utilized by a medical expert to enable a patient's health care data to be ingested by an expert network (expert system) 106 for analysis, for example, to provide a medical diagnosis. In this example, the expert network may be configured with medical experts who are trained and knowledgeable about patient symptoms of a particular medical condition or problem. This configuration may be utilized in emergency medicine, collaborative medical research, etc.

[0058] As shown in Figures 5 and 6, the input layer 108 receives tasks 103 from the task server 104. As shown in Figure 6, each expert (denoted as input nodes 1-N) in the input layer 108 receives a task 103 from the task server 104 via a corresponding device and is allowed a certain amount of time to process and submit a solution, as specified by a task time limit. Each expert in the input layer 108 provides a unique subjective solution to the task 103 as output, along with an associated confidence level. For example, a first tier expert may provide output represented as text, drawings, photographs, videos, records, calculations, predictions, computer files, and / or any other information that may constitute a solution.

[0059] Some or all of the output from each expert in the input layer 108 is aggregated into the input layer output 109 as the output of the input layer. For example, the output from each expert in the input layer 108 may be compiled into a list of solutions sorted by the confidence level provided by each expert. The input layer output 109 and the tasks 103 are then sent to the hidden layer 110. The output from one layer to the next may be sent via the task server 104 or, alternatively, directly to the experts in the subsequent layer.

[0060] As shown in FIG. 5 and FIG. 7, the hidden layer 110 receives the input layer output 109 and the tasks 103. Each node of the hidden layer 110 (labeled hidden nodes 1-N) receives (i.e., at an associated device) the input layer output 109 and the tasks 103. In some examples, the hidden layer nodes may not receive all solutions from the input layer 108. For example, solutions that are assigned a confidence level lower than the activation threshold may not be included in the input layer output 109. Each expert of the hidden layer 110 may not only select from the solutions provided in the input layer output 109, but may also assign a confidence level associated with the selection. The selection and the confidence level associated with the selection form an output from each hidden layer node. In some examples, one or more experts of the hidden layer 110 are permitted or expected to not only select multiple solutions, but also provide a confidence level associated with each selection.

[0061] Some or all of the outputs from each node of hidden layer 110 are aggregated as the output of the hidden layer into hidden layer output 111. For example, the outputs from each node of hidden layer 110 may be compiled into a list of selected solutions sorted by the confidence level provided by each hidden layer expert. The hidden layer output 111 is sent to output layer 112. In examples having multiple hidden layers, the first hidden layer output is sent to the second hidden layer for similar processing. This may continue until all hidden layers have processed tasks 103.

[0062] As shown in FIG. 5 and FIG. 8, the output layer 112 receives the hidden layer output 111 and the tasks 103. Each node of the output layer 112 (labeled as output nodes 1-N) receives the hidden layer output 111 and the tasks 103 via an associated device. In some examples, the nodes of the output layer may not receive all solutions from the hidden layer 110. For example, solutions that are assigned a confidence level lower than an activation threshold may not be included in the hidden layer output 111. Each expert of the output layer 112 may not only select from the solutions provided in the hidden layer output 111, but may also assign a confidence level associated with its selection. The selection and the confidence associated with the selection form the final output from each node of the output layer. In some examples, one or more experts in the output layer 112 may be permitted or expected to not only select multiple solutions, but also provide a confidence level associated with each selection.

[0063] Some or all of the outputs from each node in the output layer 112 are aggregated as the output of the output layer in the output layer output 113. In some examples, the confidence scores may be aggregated, i.e., the confidence scores provided by each expert for a given solution may be accumulated, averaged, and / or otherwise combined.

[0064] In some examples, the output layer output 113 comprises a single list of solutions ranked by confidence. In some examples, the output layer output 113 comprises additional listed results ranked by confidence. For example, the hidden layer output 111 may be combined with the output layer output 113.

[0065] Returning to FIG. 5, post-processing 115 may be applied to the output layer output 113 before being provided to the user 102. If utilized, post-processing 115 is configured to tailor the output of the expert network (expert system) 106 for a particular domain or specialty. In one example, post-processing 115 may assist a musical artist with determining the commercial viability of a song. In another example, post-processing 115 may assist an investor with buying, selling, and / or managing positions in stocks and / or other securities. In yet another example, post-processing 115 may inform a physician of available treatment options based on a diagnosis received from the expert network 106.

[0066] Pre-processing 105 and post-processing 115 allow the expert to further process or transform the data during network operation. For example, if the expert is analyzing music, the expert may run one or more provided software utilities to analyze or transform the audio files to complete the task.

[0067] In examples where the network is recurrent, the output layer outputs 113 are fed back to the input layer 108 along with the tasks 103 (see maximum re-input above). If the network is recurrent, the number of outputs and the resulting variation can be analyzed to determine if it is appropriate to send the final output back to the first layer.

[0068] In some instances, the expected number of final outputs is determined by max(1,[input layer expert count] / [total expert count in network]) × [input layer expert count]. If the number of final outputs is greater than the expected number and the network has not cycled more than the max re-entry setting, the network re-processes the outputs.

[0069] When outputs are fed back to the input layer 108, the original solution provided by a given node may not be fed back to that node unless additional notes, comments, references, or other data have been added or modified for review by other experts.

[0070] If the expert does not respond before the task time limit, the network can continue without the expert's response. If the expert responds at a later date, and the network is still running the iteration at that point, its output may be propagated on the next pass through the network.

[0071] After each pass through the network, the weights are adjusted and optionally stored. In some instances, nodes are rearranged as part of the competitive learning process. For example, reordering of the network layout by the task server 104 may occur after each cycle / epoch of the network.

[0072] The errors of the learning system 100 may be calculated in one or more ways depending on whether the learning system 100 is learning. See Section C below for error calculations during learning. In some examples, several error calculations may be utilized by the learning system 100 collaboratively and / or sequentially.

[0073] In some examples, errors in the learning system 100 are identified by the confidence rankings of all experts assigned to an output. If the average of all confidence scores for all outputs is less than 50% (i.e., overall confidence in the final solution is low), subtract the difference from all weights. If the average is 50% or greater, add the difference to all weights (e.g., (1-0.5)*learning rate).

[0074] In some examples, errors in the learning system 100 are identified by comparing the number of final outputs to the number of expected outputs. When the number of outputs is greater than the number of expected outputs, the network did not converge. When the network did not converge, a compensation percentage may be subtracted from all weights. Otherwise, a compensation percentage may be added to all weights.

[0075] After the weights are updated and the network is rearranged, the learning system 100 may optionally remove and replace n% of the lowest weighted experts if they are below a specified replacement rate. The replacement rate may be compared to a normalized (e.g., between 0 and 1) value of all current weights. Additionally, after the weights are updated, the weights of each expert may be updated in the expert data 122 (in some examples, multiple weights may be stored for each expert, for each unique task).

[0076] In some examples, the final network configuration is saved in network data 124. For example, one or more of the settings (e.g., expert ID, accuracy counter value, etc.) may be saved to facilitate loading and / or reconfiguring the same or a similar network again at a later point in time.

[0077] In some instances, weights are not used: instead, experts may be removed entirely and / or replaced with other experts that provide more accurate results (e.g., if the output of the original expert is erroneous).

[0078] If weights are not used, the learning rate may be utilized to determine the cutoff threshold at which an expert is replaced. In some examples, expert replacement is determined by the expert's contribution to the overall network error.

[0079] In one embodiment, a network that does not use weights uses the confidence rankings of all experts assigned to all outputs. For example, if the confidence scores of all outputs are below 50% on average, subtract the difference from all accuracy counters. If they are above 50%, add the difference to all accuracy counters. For example, (1-0.5) x learning rate.

[0080] In some examples, networks that do not utilize weights compare the number of final outputs to the number of expected outputs. If the number of outputs is greater than the number of expected outputs (i.e., the network did not converge), a percentage is subtracted from all accuracy counters, otherwise an increment is added.

[0081] After the accuracy counters are updated, each expert's accuracy counter may be used to determine if the expert should be removed from the network. The learning system 100 may optionally remove and replace n% of the expert(s) if the expert's accuracy counter falls below a replacement rate. The replacement rate is compared to the normalized (e.g., 0 to 1) values ​​of all current accuracy counters. This process is repeated for each pass through the recurrent network, allowing the learning system 100 to learn and improve on the fly, or to self-adapt without the use of weights (as in unsupervised learning).

[0082] B. Exemplary Methods for Operation of a Task Server This section describes exemplary method 200 steps for operation of a task server of the present disclosure, such as task server 104 of learning system 100 described above; see FIG. 9. Aspects of the learning system described above may be utilized in the method steps described below. Where appropriate, reference may be made to components and systems that may be used in performing each step. These references are for illustrative purposes and are not intended to limit possible ways of performing any particular step of the method.

[0083] 10 is a flow chart illustrating steps performed in an exemplary method, and may not describe a complete process or all steps of the method. Although various steps of method 200 are described below and depicted in FIG. 10, the steps need not all be performed, and may in some cases be performed simultaneously, as well as in an order different from that shown. The method provided below, in conjunction with the learning system described above, allows a user to not only submit a task, but also to receive one or more vetted candidate solutions through the propagation of expert-provided data through layers of the learning system.

[0084] Step 202 of method 200 includes receiving a user-submitted task in the learning system of the present disclosure. The task may take any form, however, it is expected that most tasks will be complex, subjective questions that require research and / or human input to answer. A task may also be an action that first requires collaboration. For example, an expert may be asked to search for a missing person or object, and / or to take photos or videos of a person, place, or object, as well as send the photos or videos to the user.

[0085] Not only should tasks have clear instructions, but if supporting electronic files are required, they should be small, easy to transmit, and virus scanned (this may occur automatically by the server of the learning system 100). Optional pre-processing may occur at this point.

[0086] A task may be composed of multiple elements or components, and the learning system 100 is configured to distribute the tasks in an efficient manner. For example, if a task is composed of 16 questions or objectives and the input layer has 8 expert nodes, two items may be sent to each expert in the input layer.

[0087] Once the task and supporting files (if any) are submitted to the learning system 100, a panel of experts (which may comprise experts from a selected network) can optionally review the task as well as provide suggestions for modifying the task if appropriate.

[0088] Step 204 of method 200 comprises selecting a group of experts for the learning system, which may be performed automatically by the learning system 100 after analyzing the task criteria, or manually at the discretion of a user who may specify selection criteria such as age, gender, location, education, income, experience, aptitude test results, previous task performance, accuracy, cost to perform the task, etc.

[0089] As part of the Expert selection process, the Experts are provided with an offer price (if specified or required) for completing the task, along with the estimated duration of the task, a full description of the task, any imposed time limits, or other relevant information. The Experts can decide whether to participate in the task. For example, each Expert can accept, reject, or counteroffer. A counteroffer can provide an alternative start time, duration, or offer price. The user or software can decide to accept or reject any counteroffers. After a selected number of Experts have accepted the invitation, or after the user has determined that a sufficient number of Experts have accepted the invitation, the process can begin. The user may be notified of the progress of the task that has begun, the estimated duration, etc., and may receive ongoing updates on the task.

[0090] As a form of control sample to help ensure that the output is reliable, one or more unqualified expert participants (based on user criteria) may be selected and added to the network. Once the experts have confirmed their interest and availability, the learning system 100 may provide test questions so that the experts can confirm their interest and ability to effectively participate in the task. In addition to the learning system 100 providing the test questions, the user may provide the test questions.

[0091] If an expert with sufficient qualifications and interest to perform the task is not available, the learning system 100 may suggest alternative and optimal times to try to re-perform the task, for example, based on the availability and do-not-disturb times set by the registered experts, or by predicting the availability of the expert based on past availability. The user can also change the selection criteria to find other experts, as well as retry the task at a later date.

[0092] Once an expert has been identified and committed to the task and the task start time is known, a message may be displayed or sent to the user indicating the estimated start time, cost, time to completion, and other information regarding the task.

[0093] Step 206 of method 200 comprises loading expert settings for the selected expert. The expert settings are loaded into a learning system (which may reside on a server and / or on a computer or device owned by the expert or user) that manages the task, including expert weights and various other settings, and may be loaded into the network layout. In some examples, a previously saved network may be loaded completely, including expert IDs, weights for each expert, confidence levels, layout, and various other settings, to facilitate loading or rebuilding the same or similar network again. In such examples, step 208 may be skipped.

[0094] Step 208 of method 200 includes determining a layout of the learning system. The learning system 100 may automatically determine an optimal layout (number of layers and number of experts per layer). In some examples, a user may specify the network layout. The network layout depends on the complexity of the task, the task elements comprising the objectives, whether learning sessions are conducted, and if so, the type and number of tasks and outputs of the learning sessions, as well as other factors.

[0095] Step 210 of method 200 comprises processing the task using the learning system (e.g., after final approval by the user). For a more detailed description of this operation, see above.

[0096] Step 212 of method 200 includes notifying the user of the results, which may include one or more outputs from the expert, as well as any accompanying files, media, data, etc.

[0097] C. Exemplary Methods for Training a Hybrid Human-Computer Learning System This section describes steps of an exemplary method 300 for training a hybrid human-computer learning system according to aspects of the present disclosure; see FIG. 10. Aspects of the learning system described above may be utilized in the method steps described below. Where appropriate, references may be made to components and systems that may be used in performing each step. These references are for illustrative purposes and are not intended to limit possible ways of performing any particular step of the method.

[0098] Training a neural network typically involves using an optimization algorithm to find an optimal set of weights. Before training a network, it is initialized, which may require creating an initial network layout, loading previously saved node weights, and modifying various settings and parameters.

[0099] There are various algorithms for training neural networks, such as backpropagation, an iterative supervised learning algorithm that calculates gradients and adjusts weights. There are also models that evolve polynomial neural networks with genetic algorithms, where the weights are evaluated and the architecture is constructed simultaneously. Self-organizing maps (Kohonen networks) and other types of neural networks may be trained in a variety of ways.

[0100] The learning system of the present disclosure may optimize its network architecture and weights in a manner that rewards nodes that contribute the most valuable and timely output to the network. When learning occurs, a training data set is labeled and knowledge is transferred to the data set so that the network learns correlations between the labels and the data.

[0101] The learning system 100, a user, and / or a third party can supply the network with a labeled training data set. The labels (also known as target values), which may be represented as text, drawings, photographs, videos, recordings, calculations, predictions, computer files, or anything else that the expert may provide as a response or output, are assigned one or more values ​​ranging from "-1" for an incorrect answer to "1" for a correct answer that specifies a known output score (other ranges may be used). The expert may be presented with correct and incorrect values ​​to choose from single or multiple options.

[0102] 10 is a flow chart illustrating steps performed in an exemplary method, and may not describe every step of a complete process or method. Although various steps of method 300 are described below and illustrated in FIG. 4, the steps need not all be performed, and may in some cases be performed simultaneously, or may be performed in an order different than that shown.

[0103] Step 302 of method 300 comprises providing a training data set to the learning system 100. The training data set (also known as target values) may be fed into the network starting at the first layer or any other layer. Tasks may be sent to experts at each layer asynchronously or synchronously, at the discretion of the user or the learning system, and may be executed on a server or on the user's computer or device.

[0104] Step 304 of method 300 comprises receiving output from a first tier expert. The first tier expert is not only presented with a task but is also allowed a certain amount of time, as specified by a task time limit user setting, to process and submit a result.

[0105] During training, experts in the first layer of the network may be only allowed to choose from one of N predefined, labeled, static outputs from the training samples (graded by correctness from -1 to 1, or by another range).

[0106] Step 306 of method 300 includes sending the results of the experts through subsequent tiers for voting. Step 308 of method 300 includes updating the expert weights as well as calculating errors in learning system 100. Errors in learning system 100 may be calculated in one or more ways. In some examples, several error calculations may be utilized collaboratively and / or sequentially by learning system 100.

[0107] For example, the difference between each expert's expected output (predicted output) and its actual output may contribute to the expert's weight during training. For example, if an expert selects an incorrect label scored as -0.5, 0.5% of the learning rate will be subtracted from the expert's weight.

[0108] Additionally or alternatively, the difference between the overall output of the learning system 100 and the expected output (i.e., the target value) may be determined for the entire network. For example, if the average of all outputs may be 0.75 for all labeled outputs, then a learning rate of 0.75% would be added to all expert weights.

[0109] Additionally or alternatively, if all confidence scores for all outputs are below 50% on average, the difference may be subtracted from all expert weights. If above 50%, the difference is added to all weights: (1-0.5)*learning rate.

[0110] Additionally or alternatively, errors in the learning system 100 may be identified by comparing the number of final outputs to the number of expected outputs. If the number of outputs is greater than the number of expected outputs, the network did not converge. If the network did not converge, a compensation percentage may be subtracted from all weights, otherwise a compensation percentage may be added to all weights.

[0111] An optional step 310 of method 300 comprises decreasing the learning rate if simulated annealing is enabled by the user and the learning rate is currently greater than 0%. The learning rate may be decreased by a set percentage of its current value (e.g., 10%). See the previous section above for more details.

[0112] An optional step 312 of the method 300 comprises sending the output from the output layer back to the input layer for further processing. If this step is performed, the method returns to step 304; if not, the method continues.

[0113] An optional step 314 of method 300 comprises sending a notification of the results to the user, such as, for example, "The network went from obtaining 10% accuracy to 80% accuracy after training."

[0114] D. Exemplary Data Processing System As shown in FIG. 6, this example describes a data processing system 600 (also referred to as a computer, computing system, and / or computer system) according to an aspect of the present disclosure. In this example, the data processing system 600 is an exemplary data processing system suitable for implementing aspects of the learning system and related methods described above. More specifically, in some examples, a device (e.g., a smartphone, a tablet, a personal computer) that is an embodiment of the data processing system may not only host data related to the learning system, execute one or more modules or software programs of the learning system, enable communication between users of the learning system and experts, and / or perform calculations and / or analyses on data obtained, provided, or generated by the learning system.

[0115] In this illustrative example, data processing system 600 includes a system bus 602 (also referred to as a communication framework). The system bus 602 may provide communication between a processor unit 604 (also referred to as a processor or multiple processors), a memory 606, persistent storage 608, a communication unit 610, an input / output (I / O) unit 612, a codec 630, and / or a display 614. The memory 606, persistent storage 608, the communication unit 610, the input / output (I / O) unit 612, the display 614, and the codec 630 are examples of resources accessible by the processor unit 604 via the system bus 602.

[0116] Processor unit 604 functions to execute instructions, which may be loaded into memory 606. Processor unit 604 may be comprised of multiple processors, multiple processor cores, and / or a particular type of processor or multiple processors (e.g., a central processing unit (CPU), a graphics processing unit (GPU), etc.), depending on the particular implementation. Additionally, processor unit 604 may be implemented using multiple heterogeneous processor systems where a main processor exists along with secondary processors on a single chip. As another illustrative example, processor unit 604 may be a symmetric multi-processor system comprising multiple processors of the same type.

[0117] Memory 606 and persistent storage 608 are examples of storage 616. Storage may comprise any suitable hardware capable of temporarily or permanently storing information (e.g., digital information), such as data, program code in a functional form, and / or other suitable information.

[0118] Storage 616 may also be referred to as a computer-readable storage device or computer-readable medium. Memory 606 may include volatile storage memory 640 and non-volatile memory 642. In some examples, a basic input / output system (BIOS), which provides the basic routines for transferring information between elements within data processing system 600, such as during start-up, may be stored in non-volatile memory 642. Persistent storage 608 may take various forms, depending on the particular implementation.

[0119] Persistent storage 608 may comprise one or more components or devices. For example, persistent storage 608 may comprise one or more devices such as a magnetic disk drive (also referred to as a hard disk drive or HDD), a solid state disk (SSD), a floppy disk drive, a tape drive, a Jaz drive, a Zip drive, a flash memory card, a memory stick, and / or the like, or any combination thereof. One or more of these devices may be removable and / or portable, for example, a removable hard drive. Persistent storage 608 may comprise one or more storage media, such as an optical disk drive, such as a compact disk ROM drive (CD-ROM), a CD recordable drive (CD-R drive), a CD rewriteable drive (CD-RW drive), and / or a digital versatile disk ROM drive (DVD-ROM), either separately or in combination with other storage media. To facilitate connection of persistent storage 608 to system bus 602, a removable or non-removable interface, such as interface 628, is typically used.

[0120] Input / Output (I / O) unit 612 allows for the input and output of data to and from other devices (i.e., input and output devices) that may be connected to data processing system 600. For example, the input devices may comprise one or more pointing devices and / or information input devices, such as a keyboard, a mouse, a trackball, a stylus, a touchpad or touchscreen, a microphone, a joystick, a gamepad, a satellite dish, a scanner, a TV tuner card, a digital camera, a digital video camera, a webcam, and / or the like. These and other input devices may be connected to processor unit 604 through system bus 602 via interface port(s). Suitable interface port(s) may include, for example, a serial port, a parallel port, a game port, and / or a universal serial bus (USB).

[0121] The output device(s) may use some of the same types of ports, and in some cases the same actual ports, as the input device(s). For example, a USB port may be used to provide input to data processing system 600 and to output information from data processing system 600 to an output device. One or more output adapters may be provided for particular output devices (e.g., monitors, speakers, printers, etc.) that require special adapters. Suitable output adapters may include, for example, video cards and sound cards that provide a means of connection between the output device and the system bus 602. Other devices, and / or systems of devices, may provide both input and output capabilities, such as remote computer(s) 660. Display 614 may include any suitable human-machine interface or other mechanism configured to display information to a user, such as, for example, a CRT, LED, or LCD monitor or screen, etc.

[0122] The communication unit 610 refers to any suitable hardware and / or software employed to provide communication with other data processing systems or devices. Although the communication unit 610 is shown internal to the data processing system 600, in some examples it may be at least partially external to the data processing system 600. The communication unit 610 may comprise internal and external technologies, such as modems (including regular telephone grade modems, cable modems, and DSL modems), ISDN adapters, and / or wired and wireless Ethernet cards, hubs, routers, and the like. The data processing system 600 may operate in a networked environment using logical connections to one or more remote computers 660. The remote computer(s) 660 may comprise a personal computer (PC), a server, a router, a network PC, a workstation, a microprocessor-based appliance, a peer device, a smartphone, a tablet, another network notebook, and / or the like. The remote computer(s) 660 typically comprise many of the elements described in conjunction with the data processing system 600. Remote computer(s) 660 may be logically connected to data processing system 600 through a network interface 662, which is connected to data processing system 600 via communication unit 610. Network interface 662 encompasses wired and / or wireless communication networks, such as local area networks (LANs), wide area networks (WANs), and cellular networks. LAN technologies may include Fiber Distributed Data Interface (FDDI), Copper Distributed Data Interface (CDDI), Ethernet, Token Ring, and / or the like. WAN technologies include point-to-point links, circuit-switched networks (e.g., Integrated Services Digital Networks (ISDN) and variations thereon), packet-switched networks, and Digital Subscriber Lines (DSL).

[0123] Codec 630 may comprise an encoder, decoder, or both, consisting of hardware, software, or a combination of hardware and software. Codec 630 may comprise any suitable device and / or software configured to encode, compress, and / or encrypt data streams or signals for transmission and storage, as well as decode, decompress, and / or decrypt data streams or signals (e.g., for video playback or editing). Although codec 630 is depicted as a separate component, codec 630 may be included in or implemented in memory, such as non-volatile memory 642.

[0124] The non-volatile memory 642 may comprise read only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, etc., or any combination thereof. The volatile storage memory 640 may comprise random access memory (RAM), which may act as external cache memory. The RAM may be configured with static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR_SDRAM), enhanced SDRAM (ESDRAM), and / or any combination thereof.

[0125] Instructions for the operating system, applications, and / or programs may be located in storage devices 616, which are in communication with processor unit 604 via system bus 602. In these illustrative examples, the instructions are in a functional form in persistent storage 608. These instructions may be loaded into memory 606 for execution by processor unit 604. The processing of one or more embodiments of the present disclosure may be performed by processor unit 604 using computer-implemented instructions, which may be located in a memory, such as memory 606.

[0126] These instructions are referred to as program instructions, program code, computer usable program code, or computer readable program code, which are executed by a processor of processor unit 604. The program code in different embodiments may be embodied on different physical or computer readable storage media, such as memory 606 or persistent storage 608. Program code 618 may be located in functional form on selectively removable computer readable medium 620 as well as loaded onto or transferred to data processing system 600 for execution by processor unit 604. Program code 618 and computer readable medium 620, in these examples, form a computer program product 622. In one example, computer readable medium 620 may comprise computer readable storage medium 624 or computer readable signal medium 626.

[0127] Computer readable storage medium 624 may comprise, for example, an optical or magnetic disk that is inserted or placed into a device, such as a drive that is part of persistent storage 608, for transfer onto a storage device, such as a hard disk that is part of persistent storage 608. Computer readable storage medium 624 may also take the form of a persistent storage medium, such as a hard drive, a thumb drive, or a flash memory, that is connected to data processing system 600. In some implementations, computer readable storage medium 624 may not be removable from data processing system 600.

[0128] In these examples, computer readable storage medium 624 is a non-transitory, physical, or tangible storage device used to store program code 618, rather than a medium that propagates or transmits program code 618. Computer readable storage medium 624 is also referred to as a computer readable tangible storage device or a computer readable physical storage device. In other words, computer readable storage medium 624 is a medium that a person can touch.

[0129] Alternatively, program code 618 may be transferred remotely to data processing system 600 using computer readable signal media 626, e.g., via a network. Computer readable signal media 626 may be, for example, a propagated data signal comprising program code 618. For example, computer readable signal media 626 may be an electromagnetic signal, an optical signal, and / or any other suitable type of signal. These signals may be transmitted over communications links, such as wireless communications links, fiber optic cable, coaxial cable, a wire, and / or any other suitable type of communications link. In other words, the communications links and / or connections may be physical or wireless in illustrative examples.

[0130] In some demonstrative embodiments, program code 618 may be downloaded over a network from another device or data processing system to persistent storage 608 via computer readable signal media 626 for use within data processing system 600. For example, program code stored on a computer readable storage medium in a server data processing system may be downloaded over a network from the server to data processing system 600. The computer providing program code 618 may be a server computer, a client computer, or other device capable of storing and transmitting program code 618.

[0131] In some examples, the program code 618 may comprise an operating system (OS) 650. The operating system 650 may be stored in persistent storage 608 and controls and allocates resources of the data processing system 600. One or more applications 652 utilize the operating system's resource management via program modules 654 and program data 656 stored in storage 616. The operating system OS 650 may comprise any suitable software system configured to manage and expose the hardware resources of the computer (600) for sharing and use by the applications 652. In some examples, the operating system OS 650 provides application programming interfaces (APIs) that facilitate the connection of different types of hardware and / or provide the applications 652 with access to hardware and operating system OS services. In some examples, a particular application 652 may provide additional services for use by other applications 652, such as in the case of so-called "middleware." Aspects of the present disclosure may be implemented in conjunction with various operating systems or combinations of operating systems.

[0132] The different components illustrated for data processing system 600 are not intended to provide architectural limitations to the manner in which different embodiments may be implemented. One or more embodiments of the present disclosure may be implemented in a data processing system including fewer components as well as in a data processing system including components in addition to and / or instead of the components illustrated for computer (600). Other components shown in FIG. 6 may vary from the depicted example. Different embodiments may be implemented using any hardware device or system capable of executing program code. As an example, data processing system 600 may include organic components integrated with inorganic components and / or may be composed exclusively of organic components (excluding humans). For example, a storage device may be composed of organic semiconductors.

[0133] In some examples, the processor unit 604 may take the form of a hardware unit having hardware circuits that are specially manufactured or configured for a particular application or to bring about a particular result or progress. This type of hardware may perform operations without requiring the program code 618 to be loaded from a storage device into a memory to be configured to perform the operations. For example, the processor unit 604 may be a circuit system, an application specific integrated circuit (ASIC), a programmable logic device, or other suitable type of hardware configured (e.g., pre-configured or reconfigured) to perform a number of operations. For example, in a programmable logic device, the device may be configured to perform a number of operations and reconfigured at a later point in time. Examples of programmable logic devices include programmable logic arrays, field programmable logic arrays, field programmable gate arrays (FPGAs), other suitable hardware devices, etc. In this type of implementation, the executable instructions (e.g., the program code 618) may be implemented as hardware, for example, by specifying an FPGA configuration using a hardware description language (HDL), and then (re)configuring the FPGA using the resulting binary file.

[0134] In another example, the data processing system 600 may be implemented as a set of FPGA-based (or possibly ASIC-based) special-purpose state machines (e.g., finite state machines (FSMs)), which may allow important tasks to be separated out and run on custom hardware. While a processor such as a CPU can be considered a shared general-purpose state machine that executes instructions provided to it, an FPGA-based state machine is purpose-built and allowed to execute hardware-coded logic without sharing resources. Such systems are often used for safety-related or mission-critical tasks.

[0135] In yet another illustrative example, processor unit 604 may be implemented using a combination of a processor and hardware units found in a computer. Processor unit 604 may include multiple hardware units and multiple processors configured to execute program code 618. In this depicted example, some of the operations may be implemented with multiple hardware units and other operations may be implemented with multiple processors.

[0136] In another example, the system bus 602 may be composed of one or more buses, such as a system bus or an input / output bus. Of course, the bus system may be implemented using any suitable type of architecture that provides for a transfer of data between different components or devices attached to the bus system. The system bus 602 may comprise several types of bus architecture(s) including a memory bus or memory controller, a peripheral bus or external bus, and / or a local bus using any of the various available bus architectures (e.g., Industry Standard Architecture (ISA), MicroChannel Architecture (MSA), Enhanced ISA (EISA), Intelligent Drive Electronics (IDE), VESA Local Bus (VLB), Peripheral Component Interconnect (PCI), CardBus, Universal Serial Bus (USB), Advanced Graphics Port (AGP), Personal Computer Memory Card International Association Bus (PCMCIA), Firewire (IEEE 1394), and Small Computer System Interface (SCSI)).

[0137] Additionally, communications unit 610 may comprise multiple devices that transmit data, receive data, or both transmit and receive data. Communications unit 610 may be, for example, a modem or a network adapter, two network adapters, or a combination thereof. Additionally, memory may be, for example, memory 606 or a cache such as found in an interface and memory controller hub that may be present on system bus 602.

[0138] E. Exemplary Distributed Data Processing System 7, this example illustrates a general network data processing system 700, which may be referred to interchangeably as a computer network, a network system, a distributed data processing system, or a distributed network, aspects of which may be included in one or more exemplary embodiments of the learning system and / or related methods described above. For example, communication between nodes of the learning system may be enabled by and / or organized as an embodiment of the network data processing system 700.

[0139] It should be appreciated that Figure 7 is provided as an illustration of one implementation and is not intended to suggest any limitation with respect to the environments in which different embodiments may be implemented. Many modifications to the depicted environments may be made.

[0140] Network data processing system 700 is a network of devices (e.g., computers) and other components, each of which may be an example of data processing system 600. Network data processing system 700 may include network 702, which is a medium configured to provide communications links between the various devices and computers connected within network data processing system 700. Network 702 may include connections such as wired or wireless communication links, fiber optic cables, and / or any other suitable medium for transmitting and / or communicating (carrying) data between network devices, or any combination thereof.

[0141] In the depicted example, a first network device 704 and a second network device 706 connect to the network 702, as well as one or more computer-readable memories or storage devices 708. The first and second network devices 704 and 706 are each an example of the data processing system 600 described above. In the depicted example, the first and second network devices 704 and 706 are shown as server computers in communication with one or more server data store(s) 722 that may be employed to store information local to the server computers (704 and 706), among other things. However, the network devices may include, but are not limited to, one or more personal computers, mobile computing devices such as personal digital assistants (PDAs) and tablets and smartphones, handheld gaming devices, wearable devices, tablet computers, routers, switches, voice gates, servers, electronic storage devices, imaging devices, media players, and / or other networked tools that may perform machine or other functions. These network devices may be interconnected through wired, wireless, optical, and other suitable communication links.

[0142] Additionally, first and second client electronic devices 710 and 712, and / or a client smart device 714, may be connected to the network 702. Each of these devices is an example of the data processing system 600 described above with respect to FIG. 6. The client electronic devices (710, 712, and 714) may comprise, for example, one or more personal computers, network computers, and / or mobile computing devices, such as personal digital assistants (PDAs), smartphones, handheld gaming consoles, wearable devices, and / or tablet computers. In the depicted example, the server 704 provides information, such as boot files, operating system images, and applications, to one or more of the client electronic devices (710, 712, and 714). The client electronic devices (710, 712, and 714) may be referred to as "clients" in the context of their relationship to a server, such as the server computer (704). The client devices may be in communication with one or more client data store(s) 720, which may be employed to store information local to the client (e.g., cookie(s) and / or associated contextual information). Network data processing system 700 may include more or fewer servers and / or clients (or no servers or clients), as well as other devices not shown.

[0143] In some examples, the first client electronic device 710 may transfer the encoded file to the server 704. The server 704 may not only store the file but also decode the file and / or transfer the file to the second client electronic device 712. In some examples, the first client electronic device 710 may transfer an uncompressed file to the server 704, which may compress the file. In some examples, the server 704 may not only encode text, audio, and / or video information, but also transmit the information to one or more clients over the network 702.

[0144] The client smart device 714 may comprise any suitable portable electronic device capable of wireless communication and software execution, such as a smartphone or tablet. In general, the term "smartphone" may refer to any suitable portable electronic device configured to perform the functions of a computer, typically having a touch screen interface, Internet access, and an operating system capable of executing downloaded applications. In addition to making phone calls (e.g., via a cellular network), a smartphone may be capable of sending and receiving email, text, and multimedia messages, accessing the Internet, and / or functioning as a web browser. A smart device (e.g., a smartphone) may comprise the functionality of other known electronic devices, such as a media player, a personal digital assistant, a digital camera, a video camera, and / or a global positioning system (GPS). A smart device (e.g., a smartphone) may be capable of wirelessly connecting with other smart devices, computers, or electronic devices, such as through near field communication (NFC), BLUETOOTH, WiFi, or a mobile broadband network. Wirelessly establishing connections between smart devices, smartphones, computers, and / or other devices may form a mobile network that allows information to be exchanged.

[0145] The data and program code located in the network data processing system 700 may be stored in or on a computer readable storage medium, such as the network connected storage devices 708 and / or the persistent storage 608 of one of the network computers, as described above, and downloaded to a data processing system or other device for use. For example, the program code may be stored in a computer readable storage medium on the server computer (704) as well as downloaded to the client (710) over the network 702 for use on the client (710). In some examples, the client data store 720 and the server data store 722 reside on one or more storage devices (708 and / or 608).

[0146] Network data processing system 700 may be implemented as one or more of different types of networks. For example, network data processing system 700 may comprise an intranet, a local area network (LAN), a wide area network (WAN), or a personal area network (PAN). In some examples, network data processing system 700 includes the Internet, where network 702 represents a worldwide collection of networks and gateways that communicate with one another using the Transmission Control Protocol / Internet Protocol (TCP / IP) suite of protocols. At the heart of the Internet is a backbone of high-speed data communication lines between major nodes or host computers. Thousands of commercial, government, educational, and other computer systems may be utilized to route data and messages. In some examples, network 702 may be referred to as a "cloud." In these examples, each server 704 may be referred to as a cloud computing node, the client electronic devices may be referred to as a cloud consumer, and so on. FIG. 7 is intended as an example and not as an architectural limitation for any illustrative embodiment.

[0147] F. Illustrative Combinations and Additional Examples This section describes additional aspects and features of the learning system, presented without limitation as a series of items (paragraphs), some or all of which may be designated alphanumeric for clarity and efficiency. Each of these items may be combined in any suitable manner with one or more other items and / or with disclosure from elsewhere in this application. Some of the following items not only explicitly refer to other items, but also further limit and provide, without limitation, some examples of suitable combinations.

[0148] (A0). A system as a data processing system for providing a solution to a task provided by a user, the system comprising a memory, one or more processors, and a plurality of instructions stored in the memory and executable by the one or more processors, the instructions comprising: receiving task-related data corresponding to a selected task, the task being identified by a selected area of ​​one or more areas of expertise; automatically selecting, from the plurality of experts, a first subset of experts associated with the selected region and a second subset of experts associated with the selected region; communicating task-related data to a plurality of first electronic devices, each of the first electronic devices being associated with a respective one of the experts in the first subset of experts; receiving, via the first electronic device, from each expert of the first subset of experts a respective task solution and an associated first confidence score; generating a first set of task solutions based on the task solutions received from the first subset of experts, the first set being sorted by the first confidence scores; communicating the task related data and the first task solution set to a plurality of second electronic devices, each of the second electronic devices being associated with one of the respective experts of the second subset of experts; receiving, via the second electronic device, information from each expert of the second subset of experts indicative of a respective selected solution selected from the first set of task solutions and an associated second confidence score; generating a set of second task solutions sorted by second confidence scores based on the information indicative of the selected solutions; generating preliminary outputs based on the second task solution set; and and communicating a final output based on the preliminary output to a user interface. system.

[0149] The instructions are further configured to automatically associate a plurality of experts with one or more domains of expertise. The system described in paragraph A0.

[0150] (A2) The said order further states: communicating the task-related data and the preliminary output to a plurality of first electronic devices; receiving, via the first electronic device, information from each expert in the first subset indicative of a respective selected solution selected from the preliminary output and an associated third confidence score; generating a third set of task solutions sorted by the third confidence scores based on information received from the first electronic device indicative of the selected solutions; communicating the task related data and the third set of task solutions to a plurality of second electronic devices; receiving, via the second electronic device, information from each expert in the second subset of experts indicative of a respective selected solution selected from the third set of task solutions and an associated fourth confidence score; generating a fourth set of task solutions sorted by the fourth confidence scores based on information indicative of the selected solutions from the second electronic device; updating the preliminary output based on at least a fourth set of task solutions before communicating the final output to the user interface. The system according to paragraph A0 or A1.

[0151] (A3) The said order further states: automatically selecting a third subset of experts classified into the selected region; communicating the task related data and the second task solution set to a plurality of third electronic devices, each third electronic device being associated with a respective one of the experts of the third expert subset; receiving, via the third electronic device, information from each expert of the third subset of experts indicative of a respective selected solution selected from the second set of task solutions and an associated third confidence score; generating a third set of task solutions sorted by the third confidence scores based on information indicative of the selected solutions received from the third electronic device; updating the preliminary output based on the third task solution set before communicating the final output to the user interface. The system according to any one of items A0 to A2.

[0152] (A4). The step of generating a set of second task solutions based on the information indicative of the selected solutions comprises sorting the set of second task solutions by an aggregation of the first confidence score and the second confidence score. The system according to any one of items A0 to A3.

[0153] (A5). Aggregating the first and second confidence scores includes calculating an average of a combination of the first and second confidence scores. The system described in paragraph A4.

[0154] (A6). The first confidence score and the second confidence score are percentages ranging from 0% to 100%; The system described in paragraph A5.

[0155] The instructions are further configured to automatically assign a respective weight to each expert; The respective weights of each expert are determined using past (historical) data on the performance of each expert. The system according to any one of items A0 to A6.

[0156] (A8) The said order further states: and being enabled to automatically remove experts from the first subset of experts and the second subset of experts having respective weights below a selected threshold. The system according to paragraph A7.

[0157] (A9). The final output has only one solution; The system according to any one of items A0 to A8. (B0) A system as a data processing system for providing a solution to a task provided by a user, The system includes a memory, one or more processors, and a number of instructions stored in the memory and executable by the one or more processors, the instructions comprising: receiving task-related data corresponding to a selected task, the selected task being identified by a selected area of ​​one or more areas of expertise; automatically selecting, from the plurality of experts, a first expert subset, one or more intermediate expert subsets, and a final expert subset, each of the first expert subset, the one or more intermediate expert subsets, and the final expert subset associated with a selected region; communicating task-related data to a plurality of first electronic devices, each of the first electronic devices being associated with a respective one of the experts in the first subset of experts; receiving, via the first electronic device, from each expert of the first subset of experts a respective task solution and an associated first confidence score; generating a set of intermediate task solutions based on the task solutions received from each expert of the first subset of experts, the intermediate task solutions being sorted by the first confidence scores; For each of said one or more intermediate subsets, successively (serially): communicating the task-related data and the intermediate task solution set to a plurality of electronic devices, each of the electronic devices being associated with one of the respective experts of each of the intermediate expert subsets; receiving, via said electronic device, information from each expert of each of said intermediate expert subsets indicative of a respective solution selected from said intermediate task solution set and an associated intermediate confidence score; and updating a set of intermediate task solutions sorted by intermediate confidence scores based on the information indicative of the selected solution; communicating the intermediate task-related data to a plurality of final electronic devices, each of the final electronic devices being associated with one of the respective experts of the final expert subset; receiving, via the final electronic device, information from each expert in the final expert subset indicative of a respective selected solution selected from the intermediate task solution set and an associated final confidence score; and and communicating the final task solution set to a user interface based on the information indicative of the selected solution. system.

[0158] (B1). The final task solution set has only a single solution; The system described in paragraph B0. (B2). The final task solution set is sorted by an aggregation of the first confidence score, the intermediate confidence score, and the final confidence score. The system according to paragraph B0 or ​​B1.

[0159] (B3). The aggregation of the first confidence score, the intermediate confidence score, and the final confidence score is determined by calculating an average of a combination of the first confidence score, the intermediate confidence score, and the final confidence score. The system according to claim B2.

[0160] (B4) The said order further states: A process for automatically assigning respective weights to each expert is implemented, The respective weights of each expert are determined using historical data on the performance of each expert. The system according to any one of items B0 to B3.

[0161] (B5). the instructions are enabled to automatically remove experts from the first subset of experts, the one or more intermediate subsets of experts, and the final subset of experts that have respective weights below a selected threshold. 4. The system according to claim B4.

[0162] The instructions are further configured to automatically pre-process the task-related data before communicating the task-related data to the plurality of first electronic devices. The system according to any one of items B0 to B5.

[0163] (B7) The step of pre-processing the task-related data includes the step of encrypting a portion of the task-related data. 4. The system according to claim B6.

[0164] (B8). The first confidence score, the intermediate confidence score, and the final confidence score are percentages ranging from 0% to 100%. The system according to any one of items B0 to B7.

[0165] (B9). One or more of the electronic devices comprises a mobile device; The system according to any one of items B0 to B8. [Benefits, Features, and Profits] Different embodiments and examples of the learning system described herein provide several advantages over known solutions for solving difficult and / or subjective problems. For example, the exemplary embodiments and examples described herein enable solutions to problems that are outside the scope of traditional computational methods.

[0166] Additionally, and among other advantages, the exemplary embodiments and examples described herein enable a network of trained experts to collectively and efficiently provide solutions to tasks posed by users.

[0167] Additionally, and among other advantages, the exemplary embodiments and examples described herein enable a network of learned experts to collectively and efficiently provide solutions to user-submitted tasks using automated peer review.

[0168] No known system or device is capable of performing these functions. However, not all of the embodiments and examples described herein provide the same advantages or to the same degree of advantages.

[0169] [Conclusion] The disclosure set forth above can encompass a number of different examples that have independent utility. Each of these has been disclosed in its preferred form(s), but since numerous variations are possible, the specific embodiments thereof disclosed and illustrated herein should not be considered in a limiting sense. To the extent that section headings are used within this disclosure, such headings are for organizational purposes only. The subject matter of the present disclosure comprises all novel and non-obvious combinations and subcombinations of the various elements, features, functions, and / or properties disclosed herein. The following claims specifically point out certain combinations and subcombinations that are deemed novel and non-obvious. Other combinations and subcombinations of features, functions, elements, and / or properties may be claimed in applications claiming priority from this or related applications. Such claims are deemed to be within the subject matter of the present disclosure regardless of whether they are broader, narrower, equal, or different than the scope of the original claims.

Claims

1. 1. A system comprising: The system includes one or more memories, one or more processors, and a plurality of instructions stored in the one or more memories and executable by the one or more processors to perform operations, the operations including: receiving task-related data corresponding to a selected task, the selected task being identified by a selected area of ​​one or more areas of expertise; automatically selecting, in real time, a first expert node subset as a first subset of a plurality of expert nodes of a learning system associated with the selected domain and a second expert node subset as a second subset of the plurality of expert nodes associated with the selected domain, the first expert node subset being associated with a first tier of a plurality of tiers of the learning system and the second expert node subset being associated with a second tier of the plurality of tiers, the selection of the first expert node subset and the selection of the second expert node subset being based on expert data stored in the plurality of expert nodes, the expert data comprising availability data for each of the plurality of expert nodes; communicating the task-related data to a plurality of first electronic devices, each of the first electronic devices being associated with one of the experts of the first expert node subset; receiving, via the first electronic device, from each expert in the first subset of expert nodes a respective task solution and an associated first confidence score; generating a first set of task solutions based on the task solutions received from the first subset of expert nodes, the first set being sorted by the first confidence scores; communicating the task-related data and the first set of task solutions to a plurality of second electronic devices, each of the second electronic devices being associated with one of the experts of the second subset of expert nodes; receiving, via the second electronic device, information from each expert in the second subset of expert nodes indicative of a respective selected solution selected from the first set of task solutions and an associated second confidence score; generating a second set of task solutions based on the information indicative of the selected solutions, the second set being sorted by the second confidence scores; generating preliminary outputs based on the second task solution set; and transmitting a final output based on the preliminary output to a user interface; Equipped with system.

2. The operations further include automatically associating the plurality of expert nodes with the one or more domains of expertise. The system of claim 1 .

3. The operation further comprises: communicating the task-related data and the preliminary output to the plurality of first electronic devices; receiving, via the first electronic device, information from each expert node of the first subset of expert nodes indicative of a respective selected solution selected from the preliminary output and an associated third confidence score; generating a third set of task solutions sorted by the third confidence scores based on the information indicative of the selected solutions received from the first electronic device; communicating the task-related data and the third set of task solutions to a plurality of second electronic devices; receiving, via the second electronic device, information from each expert node in the second subset of expert nodes indicative of a respective selected solution selected from the third set of task solutions and an associated fourth confidence score; generating a fourth set of task solutions sorted by the fourth confidence scores based on the information indicative of the selected solutions from the second electronic device; updating the preliminary output based on at least the fourth task solution set before communicating the final output to the user interface; The system of claim 1 , comprising:

4. The operation further comprises: automatically selecting a third subset of expert nodes classified into the selected region; communicating the task-related data and the second set of task solutions to a plurality of third electronic devices, each of the third electronic devices being associated with one of the expert nodes of the third expert node subset; and receiving, via the third electronic device, information from each expert node of the third subset of expert nodes indicative of a respective selected solution selected from the second set of task solutions and an associated third confidence score; generating a third set of task solutions sorted by the third confidence scores based on the information indicative of the selected solutions received from the third electronic device; updating the preliminary output based on the third task solution set before communicating the final output to the user interface; The system of claim 1 , comprising:

5. generating the second set of task solutions based on the information indicative of the selected solution comprises sorting the second set of task solutions by an aggregation of the first confidence score and the second confidence score. The system of claim 1 .

6. the aggregating the first confidence scores and the second confidence scores comprises calculating an average of a combination of the first confidence scores and the second confidence scores; the first confidence score and the second confidence score are percentages from 0% to 100%; The system of claim 5.

7. The operations further include automatically assigning a respective weight to each expert node; the respective weights of the respective expert nodes are determined using historical data regarding the performance of the respective experts; The operation further comprises: automatically removing expert nodes from the first subset of expert nodes and the second subset of expert nodes having respective weights below a selected threshold. The system of claim 1 .

8. the final output comprises only a single solution; The system of claim 1 .

9. 1. A system comprising: The system includes one or more memories, one or more processors, and a plurality of instructions stored in the one or more memories and executable by the one or more processors to perform operations, the operations including: receiving task-related data corresponding to a selected task, the selected task being identified by a selected area of ​​one or more areas of expertise; automatically selecting, in real time, a first expert node subset as a first subset of a plurality of expert nodes of a learning system, one or more intermediate expert node subsets as intermediate subsets of the plurality of expert nodes, and a final expert node subset as a final subset of the plurality of expert nodes, wherein each of the first expert node subset, the one or more intermediate expert node subsets, and the final expert node subset is associated with the selected region and a respective tier of a plurality of tiers of the learning system, and the selecting is based on expert data stored in the plurality of expert nodes, the expert data comprising availability data for each of the plurality of expert nodes; automatically assigning a respective weight to each expert node of the plurality of expert nodes, the respective weight of each expert node being determined using historical performance data of the respective expert node; communicating the task-related data to a plurality of first electronic devices, each of the first electronic devices being associated with one of the expert nodes of the first subset of expert nodes; receiving, via the first electronic device, from each expert node in the first subset of expert nodes a respective subjective response and an associated first confidence score indicative of each expert node's confidence in the subjective response; generating an intermediate task solution set sorted by the first confidence score based on the subjective responses received from each expert node of the first subset of expert nodes; For each of said one or more intermediate expert node subsets, successively: communicating the task-related data and the intermediate task solution set to a plurality of electronic devices, each of the electronic devices being associated with one of the expert nodes of each of the intermediate expert node subsets; receiving, via said electronic device, information from each expert node of each said intermediate expert node subset indicative of a respective selected solution selected from said intermediate task solution set and an associated intermediate confidence score; and updating the intermediate task solution set sorted by the intermediate confidence scores based on the information indicative of the selected solution; communicating intermediate task-related data to a plurality of final electronic devices, each of the final electronic devices being associated with one of the expert nodes of the final expert node subset; receiving, via the final electronic device, information from each expert node of the final expert node subset indicative of a respective selected solution selected from the intermediate task solution set and an associated final confidence score; and communicating a final task solution set to a user interface based on the information indicative of the selected solution; Equipped with system.

10. the final task solution set comprises only a single solution, The system of claim 9.

11. the final task solution set is sorted by an aggregation of the first confidence score, the intermediate confidence score, and the final confidence score; The system of claim 9.

12. the aggregation of the first confidence score, the intermediate confidence score, and the final confidence score is determined by calculating an average of a combination of the first confidence score, the intermediate confidence score, and the final confidence score. The system of claim 11.

13. the operations further include automatically removing expert nodes having respective weights below a selected threshold from the first expert node subset, the one or more intermediate expert node subsets, and the final expert node subset. The system of claim 9.

Citation Information

Patent Citations

  • Problem solving method

    JP1993035481A

  • Knowledge Management System for Dynamic Distributed Problem Solving

    JP2003528362A

  • Register recommended based on customer relationship management data

    JP2018513590A

  • Hybrid Human-Computer Learning System

    JP2023520309A

  • System for managing an exchange of questions and answers through an expert answer web site

    US6938068B1

Cited By

  • system

    JP2026155584A