A system for structured argumentation with asynchronous collaboration and machine-based arbitration

Abstract

A method for collecting and processing user input. In some embodiments, the method includes: presenting a prompt to a first user for eliciting a first response, the first response including a numeric portion including one or more digits, and an explanatory portion; receiving the first response from the first user; receiving a respective response from each of a plurality of other users; and displaying an ordered list of other responses to the first user. Within the ordered list, a second response from the plurality of other responses may be earlier than a third response from the plurality of other responses, the second response being further from the first response than the third response according to a measure of distance.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to and the benefit of U.S. Provisional Application No. 62 / 815,631, filed on March 8, 2019, entitled “SYSTEM OF STRUCTURED ARGUMENTATION FOR ASYNCHRONOUSCOLLABORATION AND MACHINE-BASED ARBITRATION,” the entire contents of which are incorporated herein by reference.

[0003] This application is related to U.S. Patent Application No. 16 / 591,397 filed on October 2, 2019 (“the '397 Application”), the entire contents of which are incorporated herein by reference.

[0004] Government licensing rights

[0005] This invention was made with U.S. Government support under Contract No. 2017-17061500006 awarded by the Hybrid Forecast Competition (HFC) of the Intelligence Advanced Research Projects Agency (IARPA). The U.S. Government has certain rights in this invention. Technical Field

[0006] One or more aspects of embodiments according to the present disclosure relate to forecast aggregation and arbitration. Background Art

[0007] Prior art online message boards are sometimes used by users of such message boards to request advice or opinions that may be helpful to such users. For example, a user who may want to buy or sell a certain commodity may request opinions about the likely future price of the commodity. The user may then analyze and aggregate the responses received in an ad hoc manner, attempting to assess the expertise of those who posted the responses, and extract forecasts of future prices from the responses.

[0008] This approach may have various disadvantages. On a message board, a user may start a discussion in a "thread," and other users may comment on the original post or on posts that others have sent in response to the original post. This type of discussion may lack any type of discernible structure, although responses to posts are sometimes displayed as a tree for contextual understanding. Regardless of how the structure is handled, such interfaces are not very useful for directing the discussion, which may be left entirely to the participants. Therefore, the discussion may become lopsided due to digressions, failure to cite sources to support the opinions given, or citations to sources that do not support the arguments being made.

[0009] Another potential disadvantage involves the participants themselves. Users may be victims of their biases and their initial arguments and opinions. For example, users may be affected by a type of bias called "anchoring," where a user may be strongly influenced by the first external opinion she or he sees or hears. Another potential bias is called "confirmation bias," which is a type of logical failure in which a person creates an opinion and then validates it by selecting only sources and other validation methods that agree with it, while ignoring any opinions or resources that may disagree.

[0010] Furthermore, it may be difficult or impossible for participants to reach any type of consensus, even when building consensus and reaching conclusions are the primary goals of the discussion.

[0011] Therefore, there is a need for systems and methods for facilitating productive interactions between online users. Summary of the invention

[0012] According to an embodiment of the present invention, a method for collecting and processing user input is provided, the method comprising: presenting, by a processing circuit, a prompt for inducing a first response to a first user, the first response comprising: a numerical portion comprising one or more digits, and an explanatory portion; receiving, by the processing circuit, the first response from the first user; receiving, by the processing circuit, a corresponding response from a plurality of other responses from each of a plurality of other users; displaying, by the processing circuit, an ordered list of other responses to the first user; determining, by the processing circuit, whether the first user and the other users have reached a consensus; generating, by the processing circuit, the consensus forecast in response to determining that the first user and the other users have formed a consensus forecast; generating, by the processing circuit, a coordinated forecast in response to determining that the first user and the other users have not yet formed a consensus forecast; and causing, by the processing circuit, to take an action in response to the consensus forecast or in response to the coordinated forecast, wherein: a second response from the plurality of other responses is earlier in the ordered list than a third response from the plurality of other responses, and the second response is farther from the first response than the third response according to a measure of distance.

[0013] In some embodiments, the method further includes denying the first user access to all other responses until the first response is received from the first user.

[0014] In some embodiments, the prompt for eliciting the first response includes a prompt for eliciting the numerical portion and a prompt for eliciting the explanatory portion, the prompt for eliciting the numerical portion includes a prompt for eliciting a forecast; and the prompt for eliciting the explanatory portion includes a prompt for eliciting a basic theory for the forecast.

[0015] In some embodiments, the prompt for eliciting the explanatory portion further comprises a prompt for eliciting reference documents supporting the underlying theory.

[0016] In some embodiments, the method further includes presenting to the first user a prompt for eliciting a rating of the second response.

[0017] In some embodiments, the method further includes displaying the first response to a second user of the plurality of other users, and eliciting a rating of the first response from the second user.

[0018] In some embodiments, the method further includes presenting to the second user a prompt for eliciting a rating of the referenced document in the first response.

[0019] In some embodiments, the prompt for eliciting the rating of the referenced document in the first response includes: a prompt for eliciting an assessment of the relevance of the referenced document; and a prompt for eliciting an assessment of the authenticity of the referenced document.

[0020] In some embodiments, the method further comprises, after displaying the ordered list of other responses to the first user, presenting to the first user a prompt for inducing an adjustment of the first response.

[0021] In some embodiments, the numerical portion represents an ordinal forecast and is represented by a numerical vector; and the measure of the distance for a first vector and a second vector is the ratio of: the difference between the answer density of the first vector and the answer density of the second vector; and a measure of similarity between the first vector and the second vector.

[0022] In some embodiments, the answer density for each of the first vector and the second vector is a weighted sum over the elements of the vectors, where each weight is an index of a corresponding element.

[0023] In some embodiments, the measure of similarity is the ratio of: the dot product of the vectors, and the product of the magnitudes of the vectors.

[0024] In some embodiments, the digital portion represents a classification prediction and is represented by a digital vector, and the measure of the distance of the test vector from the reference vector is the ratio of: the difference between the value of the largest element of the reference vector and the value of the element of the test vector corresponding to the largest element of the reference vector; and a measure of the similarity between the vectors.

[0025] In some embodiments, the measure of similarity is the ratio of: the dot product of the vectors, and the product of the magnitudes of the vectors.

[0026] In some embodiments, the digital portion represents a classification prediction and is represented by a digital vector, and the measure of the distance of a test vector from a reference vector is the product of: a sign term, and a preliminary distance measure of the test vector from the reference vector.

[0027] In some embodiments, the preliminary distance measure of the test vector from the reference vector is the absolute value of the ratio of: the difference between the value of the largest element of the reference vector and the value of the element of the test vector corresponding to the largest element of the reference vector; and the ratio between the dot product of the vectors and the product of the magnitudes of the vectors.

[0028] In some embodiments, the sign term is one when the value of the element of the test vector corresponding to the largest element of the reference vector is the largest element of the test vector, and negative one otherwise.

[0029] In some embodiments, the method also includes assigning, by processing circuitry, a rating to the first response and to each of the other responses.

[0030] In some embodiments, the method further includes displaying to the first user a position of the numeric portion within a range of other responsive numeric portions.

[0031] According to an embodiment of the present invention, a system for collecting and processing user input is provided, the system comprising: a processing circuit, and a memory coupled to the processing circuit and having instructions stored thereon, the instructions, when executed by the processing circuit, causing the processing circuit to: present a prompt for eliciting a first response to a first user, the first response comprising: a numeric portion comprising one or more digits, and an explanatory portion; receive the first response from the first user; receive a corresponding response from each of a plurality of other users; and display an ordered list of other responses to the first user, determine whether the first user and the other users have reached a consensus; in response to determining that the first user and the other users have formed a consensus forecast, generate the consensus forecast; in response to determining that the first user and the other users have not yet formed a consensus forecast, generate a coordinated forecast; and in response to the consensus forecast or in response to the coordinated forecast, cause an action to be taken, wherein: a second response from the plurality of other responses is earlier in the ordered list than a third response from the plurality of other responses, and the second response is farther from the first response than the third response according to a measure of distance. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] These and other features and advantages of the present disclosure will be appreciated and understood with reference to the specification, claims, and accompanying drawings, in which:

[0033] Figure 1 is a system block diagram according to an embodiment of the present disclosure;

[0034] Figure 2 is a prompt according to an embodiment of the present disclosure;

[0035] Figure 3 is a prompt according to an embodiment of the present disclosure;

[0036] Figure 4 is a prompt according to an embodiment of the present disclosure;

[0037] Figure 5A is a prompt according to an embodiment of the present disclosure;

[0038] Figure 5B is a scatter plot according to an embodiment of the present disclosure

[0039] Figure 6 is a block diagram illustrating various computerized systems in communication with each other that may be used to implement embodiments of the present invention; and

[0040] Figure 7 is a block diagram illustrating a processing system, a processing circuit, or a portion of a processing system or processing circuit for use in conjunction with at least one embodiment of the present invention. DETAILED DESCRIPTION

[0041] The specific embodiments described below in conjunction with the accompanying drawings are intended as a description of exemplary embodiments of a system for structured argumentation for asynchronous collaboration and machine-based arbitration provided in accordance with the present disclosure, and are not intended to represent the only forms in which the present disclosure may be constructed or utilized. The description is intended to illustrate features of the present disclosure in conjunction with the illustrated embodiments. However, it will be understood that the same or equivalent functions and structures may be implemented by different embodiments, which are also intended to be covered within the scope of the present disclosure. As indicated elsewhere herein, the same element numbers are intended to indicate the same elements or features.

[0042] Figure 1 A system diagram of a system for argumentation in some embodiments is shown. The system for argumentation can be configured by a user who wishes to obtain a series of opinions (or "forecasts") or consensus opinions from multiple users, referred to herein as a "host". The host may wish to poll other users for their opinions on question 105; as an illustrative example, question 105 may be to estimate (or forecast) the amount of oil that Libya will produce in May 2020. The host may configure the system according to the questions to be answered, and each participating user may then be presented with a prompt at 110 that elicits a response to the question. Figure 2 An example of a prompt that elicits a response is shown. The user is presented with a plurality of sliders 205 and a text box 210. The sliders elicit a numerical portion 215 of the response (the user's forecast of possible oil output, as discussed in further detail below), and the text box elicits an explanatory portion 220 of the response (the user's rationale for the forecast).

[0043] Each of the plurality of sliders 205 corresponds to a range or "answer interval" (the first range is values ​​less than 750, the second range is values ​​between 750 and 900, etc.). The user can adjust each slider to reflect the user's confidence that the answer is within the range corresponding to the slider. For example, if the user determines that the amount will be greater than 900 barrels (BBL) and less than 1050 barrels, the user can slide the third slider all the way to the right (indicating a 100% confidence that the answer is within this range). If, in the user's opinion, the amount is determined to be greater than 750 barrels and less than 1050 barrels, and is equally likely to (i) fall short of 900 barrels or (ii) exceed 900 barrels, the user can slide each of the second and third sliders halfway to the right, i.e., to 50%.

[0044] Software (e.g., JavaScript running in a browser in which a user interacts with the system for demonstration) may automatically adjust the sliders so that the total of the indicated four values ​​is 100%. The algorithm may prioritize adjusting sliders that the user has not yet adjusted to avoid having a slider setting change affect a previously made setting of another slider, which is not what the user wants. For example, in some embodiments, if a user has adjusted a first slider to 50%, and then attempts to adjust a second slider beyond 50% (resulting in a total > 100%), the system will simply "snap" the value of the second slider to the highest allowable value for the second slider so that their sum totals 100%. In some embodiments, once the first slider is set, the system will not change it without explicit user intervention, which may involve the user manually changing the slider, or the user clicking a button that normalizes all sliders to have equal values ​​(50% for 2 sliders, 25% for 4 sliders, etc.).

[0045] In text box 210, the user can provide a rationale for the forecast. The rationale can contain explanatory text entered by the user. In addition, because unless the participant can cite relevant and authentic sources to support their forecast, the argument may be futile, the system can provide a method for the user to cite sources that support their arguments, such as by inserting URLs in their rationales, providing URLs separately in the "reference file upload" column, or uploading documents in a network format such as Portable Document Format (pdf). These reference files can then be shown to other participants as Internet links or downloadable document files, which other participants can examine and include in their own arguments, such as by arguing specific points of the reference files or providing their own corresponding reference files. Other users can also evaluate the reference files (as discussed in further detail below).

[0046] The user can be presented Figure 2 The user may be prompted without prior or concurrent access to any forecasts that other users may have made (i.e., the user may be denied such access until a response to the prompt is received from the user); this feature (denying the user access to other users' forecasts) may eliminate anchoring and mitigate confirmation bias, e.g., reduce confirmation bias. Confirmation bias may also be mitigated by selecting the order in which other users' responses are later displayed (as discussed in further detail below).

[0047] Figure 2 The prompt of evokes a class of forecasts that may be referred to as "ordinal forecasts." As used herein, "ordinal forecasts" are those forecasts in which there is an ordering of the possible answer intervals, such as Figure 2 (each corresponding to a respective slider). Forecasts in which there is no such ordering may be called "categorical forecasts". An example of a categorical forecast is a prediction in which, for example, four candidates will win an election. In this case, the prompt may be similar to Figure 2 , where the first slider is labeled with the name of the first candidate (rather than Figure 2 The first and second sliders may be labeled "less than 750" in the example above, the second slider may be labeled with the name of the second candidate, and the like. In this forecast, the user may also use a slider to indicate a degree of confidence, e.g., if the user is certain that the second candidate will win, then adjusting the second slider all the way to the right. Whether the forecast is ordinal or categorical may affect how the "distance" between two users' forecasts (a measure of the degree to which the two users disagree) may be calculated (as discussed in further detail below). The ordinal forecast and the categorical forecast may each be represented by a vector (e.g., a vector having one element for each range or answer interval, or in the case of a categorical forecast, for each item (e.g., candidate)), each element of the vector representing the degree of user confidence that the corresponding range or item is the correct answer to the question posed by the host.

[0048] It should be understood that any other interface suitable for receiving numeric input from a user (e.g., a set of text boxes, each for receiving a corresponding number) may be used instead of a slider, and the number of input fields or controls may be different from Figure 2 The four shown, and the feature that constrains or automatically adjusts the control to have a certain total (e.g., 100%) may not exist (in which case the system can normalize the vector of digital input values ​​before further processing if appropriate). Figure 1 , after responses have been elicited (at 110) from a plurality of users, the user may provide an initial response (e.g., by responding to Figure 2 After receiving a prompt, the forecasts are ranked (at 115) and displayed to each user (at 120).

[0049] Other forecasts displayed to any user may be sorted (or "sorted" or "ranked") (at 115, Figure 1 ), where the first displayed (or "earliest") response is the forecast that is least consistent with (i.e., furthest from) the user's forecast. This allows users to review and contrast conflicting opinions; in this way, the system for argumentation can mitigate confirmation bias and drive group users to engage in self-evaluation and constructive debate. Figure 3 An example of how a forecast may be displayed to a particular user (or "current user") is shown. Summary graph 305 may show a point for each of a plurality of other users' forecasts, where the horizontal position of each point shows the distance of the other user's forecast from the current user's forecast, and the vertical position of each point shows the rating of each other user's forecast (by the other users, discussed in further detail below). A vertical line may show the horizontal position of the forecast corresponding to the current user. "Distance" may be a signed quantity, as discussed in further detail below.

[0050] Below the summary graph 305, two lists are displayed which may show forecasts for other users, with (as mentioned above) the forecasts most different from the current user's forecast being listed first, ie at the top of each list. Figure 3 The left-hand list 310 in shows forecasts that have a negative distance from the current user's forecast (other forecasts with the largest (negative) distance from the current user's forecast are shown first), and Figure 3 The right-hand list 315 in the middle right shows forecasts with a positive distance from the current user's forecast (other forecasts with the largest (positive) distance from the current user's forecast are shown first). Because some users may be accustomed to being able to change the order of the displayed lists from the default order, the system can give each user the option to search and reorder each list.

[0051] The answer intervals for each other user's forecast may be shown as a colored bar 320 at the top of each forecast (where the color in the bar corresponds to one of the answer intervals, and the length of the segment with that color corresponds to the confidence value of that answer interval). A rating bar 325 may show the number of "yes" votes received for the forecast. In some embodiments, when a rating bar 325 is shown, the rating bar 325 may show the number of "yes" votes received for the forecast. Figure 3 When prompted ( Figure 2 The prompt can be shown for example in Figure 3 ) or when it is shown Figure 4 When the prompt (discussed below) Figure 2 The prompt remains displayed. This can make it possible for the current user to adjust her or his response to the host's question, which has been influenced, for example, by the underlying theories of some other user.

[0052] The system for argumentation can be initiated by each user (at 125, Figure 1 ) Evaluation of the rationale provided by other users. When the current user hovers over (e.g., places the mouse pointer on) any other user's forecast, the current user may be presented with an opinion rating prompt, Figure 4 An example of this is shown in . Opinion rating prompt 405 may give the user the opportunity to vote “yes” by answering “yes” to the question “is this a good rationale” or to vote “no” by answering “no”. Figure 4 In an embodiment of the present invention, the small speech bubble icon 410 is a button that can open an edit field to add comments to the rationale. The actual text edit field may look similar to a box, within the borders of the opinion rating prompt 405 and below the text of the opinion rating prompt 405. The addition of text comments can be in addition to the yes / no answer to the question "Is this a good rationale?" In addition to providing visible low-effort feedback that can encourage participants to participate in the system more often, allowing users to provide an overall assessment of a given argument also provides a metric that can be used to rank and aggregate rationales.

[0053] To help track the value of user-supplied references, the system for argumentation may be triggered from each user (at 130, Figure 1) Evaluation of citations provided by other users. It may be advantageous to evaluate each citation document at least on a measure of relevance and authenticity, as these are orthogonal pieces of information for a given citation document; one reflects the relationship between the source (e.g., the cited document or publication) and the argument, and the other reflects the value of the source itself. For example, a given source may be accurate and credible, but a user relying on the citation document may have misinterpreted its meaning, causing it to be incorrectly used to support a statement that does not follow the information in the cited source. Conversely, a participant may cite a source that fully supports an argument, but the cited source may be widely understood to be untrustworthy or highly biased; in this case again, the usefulness of the citation document in the argument may be low. In particular, sources that receive multiple down votes on authenticity from subject matter experts may be removed from the discussion entirely. By eliciting feedback on relevance and authenticity, the system allows individual arguments to be weighted and judged, and promotes productive discussions by encouraging the use of external sources rather than purely opinion-based arguments.

[0054] The system used for argumentation can also trigger (at 135, Figure 1 ) User comments on the review. However, the depth of the discussion tree can be limited for such comments (for example, the system can make it impossible for users to provide comments on comments on basic theoretical reviews) to reduce the risk of the discussion going off topic.

[0055] When the current user hovers over any referenced document in another user's forecast, the current user may be presented with a prompt for initiating a rating of the referenced document. Figure 5A An example thereof is shown in FIG. Figure 5A The prompt includes: a relevance slider 505, which the current user can use to indicate the relevance of the referenced document; and an authenticity slider 510, which the current user can use to indicate the authenticity of the referenced document. Figure 5B The aggregated two-dimensional reference file evaluations are displayed in a scatter plot (e.g., to the current user, or to the host). Each reference file has a corresponding evaluation set and, for example, Figure 5B In some embodiments, users are sorted by various characteristics (e.g., by degree of expertise, such as Figure 5B The classification can be indicated in the scatter plot by, for example, displaying each point in a corresponding color.

[0056] The distance between forecasts can be defined and calculated as follows. For ordinal forecasts, the forecast can be expressed as a probability mass function p over N answer intervals, such that for each answer p i ∈[0,1] and The predicted answer density can be a weighted sum of probabilities, where the weights are based on the answer interval, for example, each weight can be the midpoint of the corresponding interval or an index of the interval. For example, the answer density can be defined as

[0057]

[0058] The similarity between two forecasts p and q is taken as their correlation,

[0059]

[0060] and the forecast distance is then defined as

[0061]

[0062] If p is a forecast for a particular user, then this metric is positive for forecasts that are more heavily weighted than p (i.e., have a higher answer density) and negative for forecasts that are less heavily weighted than p (i.e., have a lower answer density), in both cases scaled by the similarity. This allows for a meaningful partitioning of the set of forecasts into

[0063] F + ={q|D ord (p, q) ≥ 0} and F - ={q|D ord (p, q) < 0}

[0064] For categorical forecasts, distance can be defined in either of two ways. The first definition, which may be referred to as "agreement about maximum answer value," involves the degree to which two forecasts agree about the preference of one of the forecasts. That is, if user 1 assigns a maximum value to candidate B, then all other forecasts are compared by their candidate B value. The second option, which may be referred to as "agreement about maximum interval index," first involves whether the two forecasts choose the same interval as their maximum value. Details of these two definitions are provided below. In each case, p is the "reference" forecast that is compared to the "test" forecast q.

[0065] To calculate the distance based on agreement with the maximum answer value, the system may first select the interval containing the maximum value of the reference forecast,

[0066] m=argmax{p}

[0067] The system can then calculate the distance over the interval to predict the similarity scaling:

[0068]

[0069] The forecast similarity r(p,q) is defined in the same way as in the case of ordinal forecast.

[0070] If forecast q assigns a larger value to interval m, this measure is positive, which is interpreted as a more extreme agreement with forecast p. If forecast q assigns a smaller value to interval m, this measure is negative, which is interpreted as a less extreme agreement with forecast p.

[0071] To calculate the distance based on agreement on the maximum interval index, the system may select the maximum interval index for each forecast:

[0072] m=argmax{p}

[0073] n=argmax{q}

[0074] If these indices are equal, then the metric can be chosen to be positive, otherwise negative, taking the absolute value of the distance metric defined previously Scaling (used as a preliminary distance measure in the equation below):

[0075]

[0076] The symbol -1 is visible sgn|m-n| It is one when the value of the element of the test vector corresponding to the maximum element of the reference vector is the maximum element of the test vector, and negative one otherwise. This metric provides a different consensus perspective, so that forecasts on the positive side of the partition are consistent about which interval should be assigned the maximum value, and forecasts on the negative side pick some other interval for the maximum value.

[0077] In some embodiments, in addition to allowing users to interactively evaluate each other's responses and quotes through direct commentary, the system also automatically evaluates (at 140, Figure 1 ) The content and quality of each user's base theory. This method can be used to assign a score to each base theory based on the estimated accuracy of the forecast associated with the base theory, thereby providing an alternative or parallel method of judging the votes provided by users. Linguistic analysis can process the base theories and provide a set of features used to estimate the accuracy of the forecasts. In some embodiments, the extracted features are described in the '397 application. For example, various sentence structures, word usage, citation document patterns, and past forecast history of users can be related to training a supervised machine learning algorithm (e.g., support vector machine, Bayesian network or deep learning) to calculate a score, which is the probability that the forecast will be proven correct. This can help remove bias from the votes and fill in quality information in instances where there are few user votes.

[0078] This automated assessment can be implemented using a probabilistic graphical model, called probabilistic matrix factorization (PMF-RF) for rationalizing forecasts, which learns the relationship between potential user profiles and the arguments made in each forecast rationale. This method is described in the '397 application. Some embodiments of the current system for demonstration can be set to implement the PMF-RF model so that it can identify quality rationales with high probability accuracy. In some embodiments, the following model is constructed: (1) the forecasting ability of each user (that is, the frequency of the user's final demonstration being correct), (2) the difficulty of a specific problem, and (3) the discussion that occurs in response to the rationale provided by the user. By understanding the relationship between these three variables, the model predicts how accurate the user will be about each problem. In a small data setting, this model can understand the user more through the user's rationale than by only looking at its forecast accuracy. For example, if the model can extract features from the rationale, less historical data may be needed to make predictions about the forecast of a given user. This improved understanding of user ability can then be used to provide a more accurate estimate of which users may make strong arguments on either side of the debate.

[0079] In some embodiments, for example, when users of the system collaborating on an issue cannot reach a consensus, automated aggregation and coordination (at 145, Figure 1 ) to produce a coordinated solution. For example, if the change in responses at a given time (i.e., the "critical" time required to make a decision) exceeds a threshold, then this situation can be determined to exist. If the change in responses exceeds a threshold, then the system can determine that there is no agreement, and the system can use the information it already has to make a decision on behalf of the group. Another example can be the convergence of opinions over time; as consensus grows, the change can decrease toward zero (where zero change will indicate 100% agreement between all system participants). If the decrease in change toward zero is paused for a long period of time specified by the person asking the question, then the system can declare a deadlock and make a decision based on the information it already has. If the system's users collaborating on the problem cannot reach a consensus, then the system can attempt to use the feedback provided by the users on each other's basic theories and references and its own basic theory evaluation algorithm to produce a consensus opinion on the discussion issue without further input from the users. This can be done through a scoring mechanism that looks at the basic theory set of all parties in the debate and assigns a value to each option based on the weighted votes from users on each basic theory, comments on the basic theory, and the relevance and authenticity scores of the references to the basic theory. This allows the system to assign a single value score to each of the potential options for a question, which can then be normalized into a probability assignment.

[0080] For example, a multi-task modeling framework for task assignment can be used, which learns the skills of participants in the crowd and then assigns the skills to tasks based on their requirements and the skills of available participants. This framework is described, for example, in M. Rostami et al., "A Crowdsourcing Triage Algorithm for Geopolitical Event Forecasting" (RecSys'18, October 2-7, 2018, Vancouver, BC, Canada), which is incorporated herein by reference. In this method, instead of using information about the skills of the participants for assignment, the system can use the skills and feature information provided by the framework to assign weights to individual system inputs (i.e., assertions, base theory scores, and citation file scores) to produce a final aggregate system response. During the learning and parameter inference phase of this method, the system estimates the parameters of the system inputs based on historical performance (e.g., how correct the base theory provider or voter or commentator was in the past on this topic area, the past values ​​of a given reference source, etc.). This action can also incorporate authenticity and relevance scores for citations, with good referenced arguments having higher weights than arguments with poor references or arguments with no references at all. Subsequently, after learning all model parameters, the system can incorporate features of the system inputs into the aggregation process by considering the values ​​of the probabilities as described in the framework in Equation 1 of the Rostami publication identified above. Thus, a consensus opinion is created that is a weighted aggregation of the opinions of individual participants weighted by their citations and general feedback scores. For example, in some embodiments, all forecasts are averaged and the average is used as the system response. In other embodiments, a weighted average can be calculated, as described in the '397 application.

[0081] As the system performs coordination on its inputs to reach a decision, it can generate an audit trail for human evaluation to ensure that (1) the system has not made any errors in its processing, and (2) all participants can see and understand the decisions made by the system, but with full system transparency. The audit trail generated by the system can contain the following for each issue coordinated by the system: each rationale ID, each rationale score, a list of references and relevance and truth scores for each, the discussion thread structure (i.e., the ranking of each rationale or argument in the thread), and the weight of each input in the decision. A graphical user interface can be implemented to display this information to the user or it can be output as a text file.

[0082] As used herein, if the absolute value of the distance of a first forecast (of another user) from the current user's forecast exceeds the absolute value of the distance of a second forecast (of another user) from the current user's forecast, then the first forecast is "further" from the current user's forecast than the second forecast. As used herein, a "forecast" is any quantifiable information unknown to the operator because it is information related to one or more events that have not yet occurred, or for other reasons. Therefore, hypotheses about the root cause of a failure that has already occurred can be called "forecasts", even if they are not predictions about future events. As used herein, the "ratio" of a first quantity and a second quantity means the first quantity divided by the second quantity. For example, "the ratio of A and B" means A / B. As used herein, the word "or" is inclusive, so that, for example, "A or B" means any one of (i) A, (ii) B, and (iii) A and B.

[0083] Some or all of the operations described herein may be performed by one or more processing circuits. For example, a system for argumentation may be hosted on a server that includes a processing circuit, and each user and host may use a user interface (e.g., in a web browser) displayed by a computer that includes a processing circuit. The server may perform aggregation of user responses, as well as automated argument evaluation, and automated aggregation and coordination, as discussed above. The term "processing circuit" is used herein to mean any combination of hardware, firmware, and software for processing data or digital signals. The processing circuit hardware may include, for example, an application specific integrated circuit (ASIC), a general or dedicated central processing unit (CPU), a digital signal processor (DSP), a graphics processing unit (GPU), and a programmable logic device such as a field programmable gate array (FPGA). As used herein, in a processing circuit, each function is performed by hardware that is configured (i.e., hardwired) to perform the function, or by more general hardware (such as a CPU) configured to execute instructions stored in a non-temporary storage medium. The processing circuit may be manufactured on a single printed circuit board (PCB) or distributed over several interconnected PCBs. The processing circuit may include other processing circuits; for example, the processing circuit may include two processing circuits FPGA and a processing circuit CPU interconnected on a PCB.

[0084] Figure 6 is a block diagram illustrating various computerized systems in communication with each other that may be used to implement embodiments of the invention.

[0085] like Figure 6As shown, the system 800 according to some embodiments of the present disclosure is connected to a server 801 (e.g., a device to be operated) to perform the operations described herein, such as generating a report or sending a message based on a predicted state change. In this example, the server 801 may include a social media server or other electronic communication device configured to send a warning about an event to a user.

[0086] For example, system 800 may include processing circuitry configured to run an application that provides the functionality of some embodiments. This application may provide an implementation user interface (e.g., a web browser operated by each of a plurality of users) with a plurality of user interfaces (e.g., a web browser operated by each user). Figure 2-5B The application may also perform aggregation and arbitration of user responses (element 804), and it may take action (e.g., take corrective action) in response to conclusions reached (e.g., due to user consensus, or due to arbitration), as discussed in further detail below. In some embodiments, the application may be distributed, e.g., the portion of the application that provides a user interface and receives user input may run on a server different from the server that aggregates user input and may perform arbitration. The system 800 is connected to a server 801 via a network 802 to send and receive information related to various social media networks of interconnected user accounts (element 804) accessed via mobile and non-mobile devices, non-limiting examples of which include desktop computers 806, laptop computers 808, smart phones 810, and other mobile devices 812. Non-limiting examples of user accounts (element 804) include User accounts and User Accounts. As will be appreciated by those skilled in the art, a user device is any device that can receive and transmit data via network 802. Additionally, a user account may be a user account of a social media platform that may or may not be able to receive targeted marketing.

[0087] Embodiments of the invention may also be used as input to a control system. For example, a control system may be a component of a system that may also operate as a system for diagnosing the root cause of a fault (e.g., in an industrial system, a nuclear reactor, a flood control system, an automobile, or the like). In response to identifying the root cause (e.g., reaching a consensus on the root cause), the system may take corrective action to remedy the fault (e.g., in the case of a flood control system, an automated action to open or close a flood gate).

[0088] In other aspects, server 801 may be a social networking platform or advertising delivery network to access information via a social networking account (element 804) or automatically provide targeted information and / or advertisements to a display screen on a communication device (elements 806, 808, 810, 810).

[0089] Computers and other processing circuits

[0090] Various portions of embodiments of the invention that refer to the use of "processing circuitry" may be implemented with logic gates or any other embodiment of a processing unit or processing circuitry. The term "processing unit" or "processing circuitry" is used herein to include any combination of hardware, firmware, and software for processing data or digital signals.

[0091] Figure 7 is a block diagram illustrating a processing system, a processing circuit, or a portion of a processing system or processing circuit, referred to herein as a computer system, for use in connection with at least one embodiment of the present invention.

[0092] Figure 7 1 shows an exemplary computer system 1200 according to an embodiment. The exemplary computer system 1200 is configured to perform calculations, processes, operations and / or functions associated with a program or algorithm. In one embodiment, certain processes and steps discussed herein are implemented as a series of instructions (e.g., software programs) that reside in a computer-readable memory unit and are executed by one or more processing circuits of the exemplary computer system 1200. When executed, the instructions cause the exemplary computer system 1200 to perform specific actions and exhibit specific behaviors, such as those described herein.

[0093] The exemplary computer system 1200 may include an address / data bus 1210 configured to transmit information. In addition, one or more data processing units, such as processing circuits 1220, are coupled to the address / data bus 1210. The processing circuits 1220 are configured to process information and instructions. In one embodiment, the processing circuits 1220 are microprocessors. Alternatively, the processing circuits 1220 may be different types of processors, such as parallel processors, or field programmable gate arrays.

[0094] The exemplary computer system 1200 is configured to utilize one or more data storage units. The exemplary computer system 1200 may include a volatile memory unit 1230 (e.g., random access memory (“RAM”), static RAM, dynamic RAM, etc.) coupled to the address / data bus 1210, wherein the volatile memory unit 1230 is configured to store information and instructions for the processing circuit 1220. The exemplary computer system 1200 may also include a non-volatile memory unit 1240 (e.g., read-only memory (“ROM”), programmable ROM (“PROM”), erasable programmable ROM (“EPROM”), electrically erasable programmable ROM “EEPROM”), flash memory, etc.) coupled to the address / data bus 1210, wherein the non-volatile memory unit 1240 is configured to store static information and instructions for the processing circuit 1220. Alternatively, the exemplary computer system 1200 may execute instructions retrieved from an online data storage unit, such as in “cloud” computing. In one embodiment, the exemplary computer system 1200 may also include one or more interfaces, such as interface 1250, coupled to the address / data bus 1210. The one or more interfaces are configured to enable the exemplary computer system 1200 to interface with other electronic devices and computer systems. The communication interface implemented by the one or more interfaces may include wired (e.g., serial cable, modem, network adapter, etc.) and / or wireless (e.g., wireless modem, wireless network adapter, etc.) communication technology.

[0095] In one embodiment, the exemplary computer system 1200 may include an input device 1260 coupled to the address / data bus 1210, wherein the input device 1260 is configured to communicate information and command selections to the processing circuit 1220. According to one embodiment, the input device 1260 is an alphanumeric input device, such as a keyboard, which may include alphanumeric and / or function keys. Alternatively, the input device 1260 may be an input device other than an alphanumeric input device. In one embodiment, the exemplary computer system 1200 may include a cursor control device 1270 coupled to the address / data bus 1210, wherein the cursor control device 1270 is configured to communicate user input information and / or command selections to the processing circuit 1220. In one embodiment, the cursor control device 1270 is implemented using a device such as a mouse, a trackball, a trackpad, an optical tracking device, or a touch screen. Nevertheless, in one embodiment, the cursor control device 1270 is directed and / or activated via input from the input device 1260, such as in response to the use of special keys and key sequence commands associated with the input device 1260. In an alternative embodiment, cursor control device 1270 is configured to be directed or guided by voice commands.

[0096] In one embodiment, the exemplary computer system 1200 may also include one or more optional computer usable data storage devices, such as a storage device 1280, coupled to the address / data bus 1210. The storage device 1280 is configured to store information and / or computer executable instructions. In one embodiment, the storage device 1280 is a storage device such as a magnetic or optical drive, such as a hard disk drive ("HDD"), a floppy disk, a compact disk read only memory ("CD-ROM"), or a digital versatile disk ("DVD")). According to one embodiment, a display device 1290 is coupled to the address / data bus 1210, wherein the display device 1290 is configured to display video and / or graphics. In one embodiment, the display device 1290 may include a cathode ray tube ("CRT"), a liquid crystal display ("LCD"), a field emission display ("FED"), a plasma display, or any other display device suitable for displaying video and / or graphic images and user-readable alphanumeric characters.

[0097] An exemplary computer system 1200 is presented herein as an exemplary computing environment according to an embodiment. However, the exemplary computer system 1200 is not strictly limited to a computer system. For example, an embodiment provides an exemplary computer system 1200 to represent a class of data processing analysis that can be used according to various embodiments described herein. In addition, other computing systems can also be implemented. In fact, the spirit and scope of the technology of the present invention are not limited to any single data processing environment. Therefore, in one embodiment, computer executable instructions such as program modules executed by a computer are used to control or implement one or more operations of various embodiments of the technology of the present invention. In an exemplary embodiment, such program modules include routines, programs, objects, components and / or data structures configured to perform specific tasks or implement specific abstract data types. In addition, an embodiment provides one or more aspects of the technology of the present invention implemented by utilizing one or more distributed computing environments, such as where tasks are performed by remote processing devices linked by a communication network, or where various program modules are located in local and remote computer storage media including memory storage devices.

[0098] It should be understood that although the terms "first", "second", "third", etc. may be used herein to describe various elements, components, regions, layers and / or parts, these elements, components, regions, layers and / or parts should not be limited by these terms. These terms are only used to distinguish one element, component, region, layer or part from another element, component, region, layer or part. Therefore, the first element, component, region, layer or section discussed herein may be referred to as the second element, component, region, layer or section without departing from the spirit and scope of the present inventive concept.

[0099] For ease of description, spatially relative terms such as "under," "below," "below," "above," "upper," and the like may be used herein to describe the relationship of one element or feature to another element or feature, as shown in the figures. It should be understood that such spatially relative terms are intended to cover different orientations of the device in use or operation in addition to the orientation depicted in the figures. For example, if the device in the figure is turned upside down, then the elements described as "under" or "below" or "below" other elements or features will be oriented "above" the other elements or features. Therefore, the example terms "under" and "below" may cover orientations above and below. The device can be oriented in other ways (e.g., rotated 90 degrees or in other orientations), and the spatially relative descriptors used herein should be interpreted accordingly.

[0100] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the present inventive concept.

[0101] As used herein, unless the context clearly indicates otherwise, the singular form "a / an" is also intended to include the plural form. It should be further understood that when used in this specification, the term "comprises / comprising" specifies the presence of stated features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items. For example, expressions such as "at least one of" modify the entire list of elements when preceding the list of elements rather than the individual elements of the list. In addition, when describing an embodiment of the inventive concept, the use of "may" refers to "one or more embodiments of the present disclosure". In addition, the term "exemplary" is intended to refer to an example or description. As used herein, the term "use / using / used" can be considered as synonymous with the term "utilize / utilizing / utilized", respectively.

[0102] Any numerical range described herein is intended to include all subranges of the same numerical precision that fall within the range. For example, a range of "1.0 to 10.0" is intended to include subranges between (and including) the minimum value of 1.0 and the maximum value of 10.0, that is, having a minimum value equal to or greater than 1.0 and a maximum value equal to or less than 10.0, such as 2.4 to 7.6. Any maximum numerical limit described herein is intended to include all lower numerical limits contained therein and any minimum numerical limit described in this specification is intended to include all higher numerical limits contained therein.

[0103] Although exemplary embodiments of a system for structured argumentation for asynchronous collaboration and machine-based arbitration have been specifically described and illustrated herein, many modifications and variations will be apparent to those skilled in the art. Therefore, it should be understood that the system for structured argumentation for asynchronous collaboration and machine-based arbitration constructed in accordance with the principles of the present disclosure may be implemented in ways other than those specifically described herein. The invention is also defined in the appended claims and their equivalents.

Claims

1. A method for collecting and processing user input, characterized in that, The method comprises: The processing circuit presents a prompt to the first user for eliciting a first response, the first response comprising: a numeric portion, which includes one or more digits, and Explanatory section; receiving, by the processing circuit, the first response from the first user; receiving, by the processing circuit, a respective response from each of a plurality of other users; displaying, by the processing circuit, an ordered list of other responses to the first user; Determining, by the processing circuit, whether the first user and the other users have reached a consensus; In response to determining that the first user and the other users have formed a consensus forecast, generating, by the processing circuit, the consensus forecast; In response to determining that the first user and the other users have not formed a consensus forecast, generating, by the processing circuitry, a coordinated forecast; and causing, by the processing circuitry, a control system to take an automated action in response to the consensus forecast or in response to the coordinated forecast, in: A second response in the plurality of other responses is earlier in the ordered list than a third response in the plurality of other responses, and The second response is further from the first response than the third response is according to a measure of distance.

2. The method according to claim 1, characterized in that Also included is denying the first user access to all other responses until the first response is received from the first user.

3. The method according to claim 1, characterized in that the prompt for eliciting the first response includes a prompt for eliciting the numeric portion and a prompt for eliciting the explanatory portion, said prompt for initiating said digital portion comprises a prompt for initiating a forecast; and The prompt for eliciting the explanatory portion includes a prompt for eliciting a rationale for the forecast.

4. The method according to claim 3, characterized in that The prompt for eliciting the explanatory section also includes a prompt for eliciting references supporting the underlying theory.

5. The method according to claim 1, characterized in that Also included is presenting a prompt to the first user for eliciting a rating of the second response.

6. The method according to claim 1, characterized in that Also includes: displaying the first response to a second user of the plurality of other users, and A rating of the first response is elicited from the second user.

7. The method according to claim 6, characterized in that Also included is presenting a prompt to the second user for eliciting a rating of the referenced document in the first response.

8. The method according to claim 7, characterized in that The prompt for inducing the rating of the referenced document in the first response includes: a prompt for initiating an evaluation of the relevance of the referenced document; and A prompt for evaluating the authenticity of the cited document.

9. The method according to claim 1, characterized in that: Also included is after displaying the ordered list of other responses to the first user, A prompt is presented to the first user for inducing an adjustment of the first response.

10. The method according to any one of claims 1 to 9, characterized in that: The numeric portion represents an ordinal forecast and is represented by a numeric vector; and The distance measure is the ratio of the first vector to the second vector: The difference between: The answer density of the first vector, and the answer density of the second vector; and A measure of similarity between the first vector and the second vector.

11. The method according to claim 10, characterized in that The answer density for each of the first vector and the second vector is a weighted sum over the elements of the vectors, where each weight is an index of a corresponding element.

12. The method according to claim 11, characterized in that The measure of similarity is the ratio of: The dot product of the vectors, and The product of the magnitudes of the vectors.

13. The method according to any one of claims 1 to 9, characterized in that: The numeric part represents the classification forecast and is represented by a numeric vector, The measure of the distance of the test vector from the reference vector is the ratio of: The difference between: the value of the largest element of the reference vector, and the value of the element of the test vector corresponding to the largest element of the reference vector; and A measure of the similarity between the vectors.

14. The method according to claim 13, characterized in that The measure of similarity is the ratio of: The dot product of the vectors, and The product of the magnitudes of the vectors.

15. The method according to any one of claims 1 to 9, characterized in that: The numeric part represents the classification forecast and is represented by a numeric vector, The measure of the distance of the test vector from the reference vector is the product of: Signed terms, and A preliminary distance measure of the test vector from the reference vector.

16. The method according to claim 15, characterized in that The preliminary distance measure of the test vector from the reference vector is the absolute value of the ratio of: The difference between: the value of the largest element of the reference vector, and the value of the element of the test vector corresponding to the largest element of the reference vector; and The ratio between: The dot product of the vectors, and The product of the magnitudes of the vectors.

17. The method according to claim 16, characterized in that The sign terms are: is one when the value of the element of the test vector corresponding to the largest element of the reference vector is the largest element of the test vector, and Otherwise it is negative one.

18. The method according to any one of claims 1 to 9, characterized in that: Also included is assigning, by processing circuitry, a rating to the first response and to each of the other responses.

19. The method according to any one of claims 1 to 9, characterized in that: Also included is displaying to the first user the position of the numeric portion within a range of other responsive numeric portions.

20. A system for collecting and processing user input, characterized in that The system comprises: processing circuitry, and a memory coupled to the processing circuit and having stored thereon instructions that, when executed by the processing circuit, cause the processing circuit to: Presenting a prompt to a first user for eliciting a first response, the first response comprising: a numeric portion, which includes one or more digits, and Explanatory section; receiving the first response from the first user; receiving a corresponding response from each of the plurality of other users; and displaying an ordered list of other responses to the first user; Determining whether the first user and the other users have reached a consensus; In response to determining that the first user and the other users have formed a consensus forecast, generating the consensus forecast; In response to determining that the first user and the other users have not yet formed a consensus forecast, generating a coordinated forecast; and causing a control system to take an automated action in response to the consensus forecast or in response to the coordinated forecast, in: A second response in the plurality of other responses is earlier in the ordered list than a third response in the plurality of other responses, and The second response is further from the first response than the third response is according to a measure of distance.

Images